Image sensor and image capturing apparatus

ABSTRACT

An image sensor comprises a first imaging pixel and a second imaging pixel each of which detects an object image formed by a photographing optical system and generates a recording image. Each of the first imaging pixel and the second imaging pixel comprises a plurality of photoelectric conversion units segmented in a first direction, the plurality of photoelectric conversion units have an ability of photoelectrically converting images formed by split light beams out of a light beam from the photographing optical system and outputting focus detection signals to be used to detect a phase difference. A base-line length of photoelectric conversion units to be used to detect the phase difference included in the first imaging pixel is longer than that of photoelectric conversion units to be used to detect the phase difference included in the second imaging pixel.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 13/476,161,filed May 21, 2012, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image sensor and an image capturingapparatus having the image sensor and, more specifically, to an imagesensor capable of capturing a still image and/or a moving image using anumber of photoelectric conversion units that are arrangedtwo-dimensionally, and an image capturing apparatus for performing phasedifference focus detection using the image sensor.

2. Description of the Related Art

For an electronic camera capable of recording a moving image or a stillimage, there has been proposed a technique of implementing focusdetection by a phase difference detection method using an image sensorfor image recording. In the phase difference detection method, a lightbeam that has passed through the exit pupil of a photographing opticalsystem is split into two light beams. The two split light beams arereceived by a set of light-receiving element groups for focus detection.The shift amount between the signal waveforms of a pair of images whichare two images output in accordance with the light receiving amounts,that is, the relative positional shift amount generated in the pupildivision direction of the light beam is detected, thereby obtaining thefocus shift amount (defocus amount) of the photographing optical system.The focus detection characteristic of this method depends on the arrayof the focus detection pixels or the pupil division characteristicshapes of the pixels. Hence, various techniques have been proposedconcerning the intra-pixel structure or array to improve the focusdetection characteristic.

On the other hand, the image sensor is anticipated to acquire ahigh-resolution image containing little noise. For this purpose, eachpixel of the image sensor preferably receives a light beam having passedthrough a region as wide as possible in the exit pupil of thephotographing optical system. However, using the light beam in the widepupil region may conflict with improving the performance in phasedifference focus detection. To satisfy both the image capturing abilityand the phase difference detection ability, the following techniqueshave been proposed.

In, for example, Japanese Patent Laid-Open No. 2007-158692, each pixelof the image sensor has a first photoelectric conversion unit arrangedin the central region of the pixel and a second photoelectric conversionunit arranged around it. An image signal is generated using the outputof the first photoelectric conversion unit, and phase difference focusdetection is performed using the output of the second photoelectricconversion unit.

In Japanese Patent Laid-Open No. 2009-015164, a plurality of pixelgroups having different split center positions of photoelectricconversion units are provided to ensure redundancy for a change in theexit pupil position of the photographing optical system. An optimumpixel group is selected in accordance with the exit pupil position,thereby reducing unbalance of the light receiving amount of the focusdetection signal.

In Japanese Patent Laid-Open No. 2007-279312, two types of focusdetection pixels are provided independently of imaging pixels. Adistance w3 between the gravity centers of distance measurement pupilsin the pupil arrangement direction of one type of focus detection pixelsis made different from a distance w3 between the gravity centers ofdistance measurement pupils in the pupil arrangement direction of theother type of focus detection pixels. There is disclosed selecting oneof the two types of focus detection pixels based on the magnitude of thedefocus amount.

However, in the technique disclosed in Japanese Patent Laid-Open No.2007-158692, since the pixel arrangement emphasizes the image capturingcharacteristic, a satisfactory focus detection characteristic is notnecessarily obtained. For example, for a photographing optical systemhaving a large f-number, that is, a small exit pupil diameter, the lightbeam to the photoelectric conversion unit for focus detection isvignetted, and focus detection may be impossible. In addition, in theperipheral portion of the image sensor, that is, in the region with alarge image height, the exit pupil diameter becomes small due tovignetting of the photographing optical system. The vignetting statechanges depending on the model of the photographing optical system, thezoom state and focus state. Hence, a focus-detectable region alsochanges depending on these states, making stable focus detectiondifficult.

In the technique disclosed in Japanese Patent Laid-Open No. 2009-015164,since pupil division is limited in one direction, focus detection cannotbe performed for an object having a luminance distribution only in adirection orthogonal to it. To increase the device sensitivity for imageacquisition, the area of the photoelectric conversion unit needs to belarge. However, in a large defocus state, the blur of the focusdetection image also becomes large, and the focus-detectable defocusrange narrows.

In Japanese Patent Laid-Open No. 2007-279312, the two types of focusdetection pixels do not serve as imaging pixels and therefore becomedefective pixels when acquiring an image.

On the other hand, when an image sensor having a pupil division abilityis used, a 3D image having parallax information can be acquired.However, how to optimize both the focus detection ability and the 3Dimage acquisition ability is unknown even when the techniques describedin Japanese Patent Laid-Open Nos. 2007-158692 and 2009-015164 are used.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and widens a focus-detectable defocus range and improves thedetection accuracy in a near in-focus state.

The present invention further widens a focus-detectable defocus rangeand improves the detection accuracy in a near in-focus stateindependently of the direction of the luminance distribution of anobject.

The present invention further optimizes both a focus detection abilityand a 3D image acquisition ability.

According to the present invention, provided is an image sensorcomprising: a first imaging pixel and a second imaging pixel each ofwhich detects an object image formed by a photographing optical systemand generates a recording image, wherein each of the first imaging pixeland the second imaging pixel comprises a plurality of photoelectricconversion units segmented in a first direction, the plurality ofphotoelectric conversion units have an ability of photoelectricallyconverting a plurality of images formed by split light beams out of alight beam from the photographing optical system and outputting focusdetection signals to be used to detect a phase difference, and abase-line length of photoelectric conversion units to be used to detectthe phase difference out of the plurality of photoelectric conversionunits included in the first imaging pixel is longer than a base-linelength of photoelectric conversion units to be used to detect the phasedifference out of the plurality of photoelectric conversion unitsincluded in the second imaging pixel.

Further, according to the present invention, provided is an image sensorcomprising: an imaging pixel which detects an object image formed by aphotographing optical system and generates a recording image, whereinthe imaging pixel comprises 2n (n is an integer not less than 2)photoelectric conversion units divided in a first direction, each of the2n photoelectric conversion units has an ability of photoelectricallyconverting an image formed by a split light beam out of a light beamfrom the photographing optical system and outputting a focus detectionsignal to be used to detect a phase difference, and the image sensor hasa mode in which the phase difference is detected using two photoelectricconversion units having a small base-line length and arranged inside outof the 2n photoelectric conversion units included in the imaging pixeland a mode in which the phase difference is detected using twophotoelectric conversion units having a large base-line length andarranged outside the two photoelectric conversion units having the smallbase-line length out of the 2n photoelectric conversion units includedin the imaging pixel.

Furthermore, according to the present invention, provided is an imagecapturing apparatus including one of the foregoing image sensors.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the schematic arrangement of an imagecapturing apparatus according to a first embodiment of the presentinvention;

FIG. 2 is a view showing the pixel array of an image sensor according tothe first embodiment;

FIG. 3 is a view showing the arrangement of the read circuit of theimage sensor according to the first embodiment;

FIGS. 4A and 4B are explanatory views of the projection relationship ofthe photoelectric conversion units of pixels at the center of the screenwhen the zoom state is the Middle state;

FIGS. 5A to 5D are views showing the projected images of the pixels atthe center of the screen on the exit pupil plane when the zoom state isthe Middle state;

FIGS. 6A and 6B are explanatory views of the projection relationship ofthe photoelectric conversion units of pixels at the periphery of thescreen when the zoom state is the Middle state;

FIGS. 7A to 7D are views showing the projected images of the pixels atthe periphery of the screen on the exit pupil plane when the zoom stateis the Middle state;

FIGS. 8A and 8B are explanatory views of the projection relationship ofthe photoelectric conversion units of pixels at the periphery of thescreen when the zoom state is the Wide state;

FIGS. 9A and 9B are explanatory views of the projection relationship ofthe photoelectric conversion units of pixels at the periphery of thescreen when the zoom state is the Tele state;

FIGS. 10A and 10B are views for explaining the arrangement of thephotoelectric conversion units of the pixels at specific positions onthe screen;

FIG. 11 is a view for explaining the positional relationship between thephotoelectric conversion units and the projected images of the exitpupil when the zoom state is the Middle state;

FIG. 12 is a view for explaining the positional relationship between thephotoelectric conversion units and the projected images of the exitpupil when the zoom state is the Wide state;

FIG. 13 is a view for explaining the positional relationship between thephotoelectric conversion units and the projected images of the exitpupil when the zoom state is the Tele state;

FIGS. 14A to 14C are graphs for explaining changes in the output signalsof a first pixel and a second pixel caused by defocus;

FIGS. 15A and 15B are graphs for explaining the difference in the outputsignal between the first pixel and the second pixel depending on thefocus detection area;

FIGS. 16A to 16C are conceptual views for explaining a method of addingthe outputs of photoelectric conversion units when creating a 3D imageaccording to the first embodiment;

FIGS. 17A and 17B are views showing examples of an image, focusdetection signals, and a defocus map at the time of focus detection;

FIG. 18 is a flowchart showing the main procedure of a camera at thetime of photographing according to the first embodiment;

FIG. 19 is a flowchart of a focus detection subroutine according to thefirst embodiment;

FIG. 20 is a flowchart of an image recording subroutine according to thefirst embodiment;

FIG. 21 is a view showing the pixel array of an image sensor accordingto a second embodiment;

FIGS. 22A to 22C are conceptual views for explaining a method of addingthe outputs of photoelectric conversion units when creating a 3D imageaccording to the second embodiment;

FIG. 23 is a view showing the pixel array of an image sensor accordingto a third embodiment;

FIG. 24 is a flowchart showing the main procedure of a camera at thetime of photographing according to a fourth embodiment;

FIG. 25 is a flowchart of a focus detection subroutine according to thefourth embodiment;

FIGS. 26A and 26B are views for explaining a plurality of base-linelengths of an image sensor according to a fifth embodiment;

FIGS. 27A and 27B are tables showing weighting coefficients for threekinds of defocus amounts according to the fifth embodiment;

FIG. 28 is a flowchart of a focus detection subroutine according to thefifth embodiment;

FIG. 29 is a table showing weighting coefficients for three kinds ofdefocus amounts according to a modification of the fifth embodiment;

FIG. 30 is a table showing weighting coefficients for three kinds ofdefocus amounts according to another modification of the fifthembodiment;

FIG. 31 is a table showing weighting coefficients for three kinds ofdefocus amounts according to a sixth embodiment;

FIG. 32 is a flowchart of a focus detection subroutine according to thesixth embodiment; and

FIG. 33 is a table showing weighting coefficients for three kinds ofdefocus amounts according to a modification of the sixth embodiment.

DESCRIPTION OF THE EMBODIMENTS

The best mode for carrying to the present invention will now bedescribed in detail below with reference to the accompanying drawings.

First Embodiment

FIG. 1 shows the schematic arrangement of a digital camera serving as animage capturing apparatus including an image sensor according to thepresent invention. FIG. 1 illustrates a digital camera formed byintegrating or connecting a camera body including an image sensor and aphotographing optical system. The digital camera can record movingimages and still images. Referring to FIG. 1, a first lens group 101 isarranged at the end of the photographing optical system (imaging opticalsystem) and held such that it can move reciprocally in the optical axisdirection. A stop 102 adjusts its aperture diameter to adjust the lightamount at the time of photographing, and also has an ability as anexposure time control shutter when photographing a still image.Reference numeral 103 denotes a second lens group. The stop 102 and thesecond lens group 103 integrally move reciprocally in the optical axisdirection in synchronism with the reciprocal movement of the first lensgroup 101, thereby implementing a scaling ability (zoom ability).

A third lens group 105 moves reciprocally in the optical axis directionto adjust focus. An optical low-pass filter 106 is an optical elementthat reduces the false color or moiré of a captured image. An imagesensor 107 is formed from a two-dimensional CMOS sensor and peripheralcircuits thereof. The image sensor 107 uses a two-dimensional singlemulticolor sensor in which M horizontal pixels and N vertical pixels arearranged in a matrix, and on-chip primary color mosaic filters areformed in a Bayer arrangement. Note that the arrangement of the imagesensor 107 will be described later in detail.

A zoom actuator 111 makes a cam tube (not shown) rotate manually or byan actuator so as to reciprocally move the first lens group 101 to thethird lens group 105 in the optical axis direction, thereby performingthe scaling operation. A stop shutter actuator 112 controls the aperturediameter of the stop 102 to adjust the photographing light amount andalso controls the exposure time when photographing a still image. Afocus actuator 114 reciprocally moves the third lens group 105 in theoptical axis direction to adjust focus.

A wireless communication unit 115 includes an antenna and a signalprocessing circuit to communicate with a server computer via a networksuch as the Internet. An attitude detection unit 116 of the camera usesan electronic level to determine the photographing attitude of thecamera, that is, landscape orientation photographing or portraitorientation photographing.

An intra-camera CPU 121 performs various kinds of control of the camerabody and includes an arithmetic unit, a ROM, a RAM, an A/D converter, aD/A converter, a communication interface circuit, and the like. The CPU121 drives various kinds of circuits provided in the camera and executesa series of operations including AF, photographing, and image processingand recording, and the like based on a predetermined program stored inthe ROM.

A communication control circuit 122 transmits a captured image from thecamera to the server computer via the communication unit 115 or receivesan image or various kinds of information from the server computer. Anattitude detection circuit 123 determines the attitude of the camerabased on the output signal of the attitude detection unit 116. An imagesensor driving circuit 124 controls the image capturing operation of theimage sensor 107, and also A/D-converts an acquired image signal andtransmits it to the CPU 121. An image processing circuit 125 performsprocessing such as γ conversion, color interpolation, and JPEGcompression of an image acquired by the image sensor 107.

A focus driving circuit 126 drives and controls the focus actuator 114based on a focus detection result so as to reciprocally drive the thirdlens group 105 in the optical axis direction, thereby adjusting focus. Astop driving circuit 128 drives and controls the stop shutter actuator112 to control opening of the stop 102. A zoom driving circuit 129drives the zoom actuator 111 in accordance with the zoom operation ofthe user.

A display device 131 such as an LCD displays information concerning thephotographing mode of the camera, a preview image before photographingand an image for check after photographing, an in-focus state displayimage upon focus detection, attitude information of the camera, and thelike. An operation switch group 132 includes a power switch, aphotographing start switch, a zoom operation switch, a photographingmode selection switch, and the like. A detachable flash memory 133records captured images.

FIG. 2 shows the pixel array of the image sensor 107 according to thefirst embodiment, which is manufactured using the technique disclosed inJapanese Patent Laid-Open No. 09-046596 applied by the present inventor.FIG. 2 shows a state in which a range of 12 rows in the verticaldirection (Y direction) and 14 columns in the horizontal direction (Xdirection) of a two-dimensional CMOS area sensor is observed from thephotographing optical system side. The Bayer arrangement is applied tothe color filters. Green and red color filters are alternately providedon the pixels of the odd-numbered rows. Blue and green color filters arealternately provided on the pixels of the even-numbered rows. A circle211 i represents an on-chip microlens. Each of a plurality of rectanglesarranged inside the on-chip microlens 211 i represents a photoelectricconversion unit.

In the present invention, every pixel includes a plurality ofphotoelectric conversion units segmented into m parts in the X directionand n parts in the Y direction (m and n are integers of 1 or more), andphotoelectric conversion signals of the photoelectric conversion unitscan be read independently. It should noted that the segmentationpatterns of photoelectric conversion units in pixels are not the same,and the image sensor includes a plurality of pixels having differentsegmentation patterns. The features of these pixels will be describedbelow. Note that in the following explanation, the shape of a pluralityof segmented photoelectric conversion units which are connected andregarded as one photoelectric conversion unit will be referred to as aconnected shape, and the center of the connected shape will be referredto as a connection center hereinafter.

A first pixel 211 has a total of four photoelectric conversion units 211a to 211 d segmented into two parts (integer m1=2) in the X directionand two parts (integer n1=2) in the Y direction. The four photoelectricconversion units 211 a to 211 d are segmented to have line symmetryabout the X- and Y-axes passing through the pixel center. That is, eachof the segmented photoelectric conversion units has a square planarshape. The connected shape of the four photoelectric conversion units isalso square. The first pixel 211 have the same segmented shape at allpositions on the image plane. The outputs of the first pixel 211 areused for recording image generation and focus detection in a nearin-focus state. Recording images include not only a normal 2D(2-Dimensional) image defined by a format such as JPEG but also a 3D(3-Dimensional) image formed from a plurality of images having parallaxinformation. Both a moving image and a still image are included. Notethat the other pixels having the same structure and arranged in theimage sensor 107 will also be referred to as the first pixels 211hereinafter.

Second pixels 221 to 223 are discretely arranged among the first pixels211 in accordance with a predetermined array rule. Each of the secondpixels has a total of four photoelectric conversion units segmented intofour parts (integer m2=4) in the X direction but not segmented (integern2=1) in the Y direction. The connected shape of the photoelectricconversion units is square in the second pixels 221 to 223 as well. Theoutputs of the second pixels 221 to 223 are used for recording imagegeneration and focus detection in a large defocus state (when the focusshift amount is large). In FIG. 2, out of the four segmentedphotoelectric conversion units, the two photoelectric conversion units(indicated by suffixes b and c) at the center are used for focusdetection. Referring to FIG. 2, the focus detection photoelectricconversion units are indicated as solid parts, although their basicstructure and characteristics are the same as those of the photoelectricconversion units (indicated by suffixes a and d) on both sides. Notethat the pixels having the same structure and arranged in the imagesensor 107 will also be referred to as second pixels 221 to 223hereinafter. The connected shape of the four photoelectric conversionunits is square even in the second pixels 221 to 223. However, thesecond pixels 221 to 223 are further classified into three types by theplanar shape difference between the segmented individual photoelectricconversion units, as will be described below.

In the second pixel 222, photoelectric conversion units 222 b and 222 care arranged to be bilaterally symmetrical about the pixel center. Thatis, the connection center of photoelectric conversion units 222 a to 222d matches that of the photoelectric conversion units 222 b and 222 c atthe center. The X-direction size of the photoelectric conversion units222 b and 222 c (the width of the photoelectric conversion units) is setto be smaller (narrower) than the X-direction size of the outerphotoelectric conversion units 222 a and 222 d. This segmented shape iscommon to all second pixels 222 arranged on the image plane.

A composition of the outputs of the photoelectric conversion units 222 bof the second pixels 222 arranged on the same row within a predeterminedrange is defined as a B image for AF, and a composition of the outputsof the photoelectric conversion units 222 c is defined as a C image forAF. The relative shift amount between the B image and the C image for AFis detected by correlation, thereby detecting the focus shift amount,that is, the defocus amount in the predetermined area.

In the second pixel 221, the connection center of photoelectricconversion units 221 b and 221 c is shifted in the negative direction onthe X-axis with respect to that of photoelectric conversion units 221 ato 221 d. However, the X-direction size of the photoelectric conversionunits 221 b and 221 c is set to be equal to that of the photoelectricconversion units 222 b and 222 c. As a result, the X-direction size ofthe outer photoelectric conversion unit 221 a is smaller than that ofthe photoelectric conversion unit 222 a. The X-direction size of theother photoelectric conversion unit 221 d is larger than that of thephotoelectric conversion unit 222 d.

In the second pixel 223, the connection center of photoelectricconversion units 223 b and 223 c is shifted in the positive direction onthe X-axis with respect to that of photoelectric conversion units 223 ato 223 d. However, the X-direction size of the photoelectric conversionunits 223 b and 223 c is set to be equal to that of the photoelectricconversion units 222 b and 222 c. As a result, the X-direction size ofthe outer photoelectric conversion unit 223 a is larger than that of thephotoelectric conversion unit 222 a. The X-direction size of the otherphotoelectric conversion unit 223 d is smaller than that of thephotoelectric conversion unit 222 d.

The reason why the photoelectric conversion units 221 b, 221 c, 222 b,222 c, 223 b, and 223 c of the second pixels 221, 222, and 223 aresmaller in the X direction is as follows. In the phase differencedetection type focus detection system, pupil division of a focusdetection light beam is done on the exit pupil of the photographingoptical system. If the pupil size in the pupil division direction islarge, the blur of an AF image in a non-focus state is large, and thefocus detectable range becomes narrow, that is, the focus detectioncapability in a large defocus state degrades. In addition, when thef-number of the photographing optical system is large, the focusdetection light beam is conspicuously vignetted. This degrades thesimilarity between a pair of AF image signals or increase the unbalanceof the light amount. Since this vignetting phenomenon depends on thedefocus amount, the focus detection ability in the large defocus statefurther degrades.

The focus detection pupil on the exit pupil of the photographing opticalsystem and the photoelectric conversion units of each pixel of the imagesensor have a conjugated relation through the on-chip microlenses.Hence, making the photoelectric conversion units 221 b, 221 c, 222 b,222 c, 223 b, and 223 c to be used for focus detection smaller in the Xdirection allows to narrow the width of the focus detection pupil andthus avoid the decrease in the focus detection performance in the largedefocus state.

On the other hand, in a near in-focus state, that is, when the defocusamount is small, the blur of the image is small even if the size of thefocus detection pupil is large. Hence, upon determining a near in-focusstate, the outputs of the first pixels 211 are also used for focusdetection. This enables to increase the information amount of the AFimage signal to be used for focus detection, reduce the influence ofnoise of the pixel outputs, and thus improve the focus detectionaccuracy. Details will be described later.

The above-described second pixels 221 to 223 are pixels for pupildivision in the X direction and are used for an object having aluminance distribution in the X direction, that is, an object having avertical-striped pattern. To the contrary, third pixels 224 to 226 arepixels for pupil division in the Y direction and are used for an objecthaving a luminance distribution in the Y direction, that is, an objecthaving a lateral-striped pattern. The third pixels 224 to 226 are alsosegmented into four photoelectric conversion units in the Y direction.The third pixels are further classified into three types in accordancewith the segmented shape of the photoelectric conversion units. Thesepatterns are equivalent to those of the second pixels 221 to 223 rotatedby 90°, and a detailed description thereof will be omitted. A suitableone of the second pixels 221 to 223 and the third pixels 224 to 226 isused in accordance with the luminance distribution pattern of a focusdetection target object, thereby reducing the probability that focusdetection is impossible.

FIG. 3 shows the arrangement of the read circuit of the image sensor 107according to the present invention. Reference numeral 151 denotes ahorizontal scanning circuit; and 153, a vertical scanning circuit.Horizontal scanning lines 152 a to 152 d and vertical scanning lines 154a to 154 d are arranged at the boundaries of the pixels. The signalsfrom the photoelectric conversion units are externally read throughthese scanning lines.

Note that the image sensor 107 of the present invention has thefollowing two read modes. The first read mode is called a progressivescanning mode to be used to capture a high-resolution still image. Inthis case, the signals of all pixels are read. The second read mode iscalled a down sampling mode to be used to record a moving image or onlydisplay a preview image. In this case, since the number of necessarypixels is smaller than the total number of pixels, the first pixels 211down-sampled at a predetermined ratio in both the Y and X directions areread. The focus detection ability is maintained by reading all thesecond pixels 221 to 223 and the third pixels 224 to 226.

FIGS. 4A and 4B are views for explaining the conjugated relation betweenthe exit pupil plane of the photographing optical system andphotoelectric conversion units at an image height of 0, that is,arranged near the center of the image plane of the image sensor in thecamera of the present invention. The photoelectric conversion units inthe image sensor 107 and the exit pupil plane of the photographingoptical system are designed to have a conjugated relation through theon-chip microlenses 211 i. In general, the exit pupil plane of thephotographing optical system almost matches the plane on which the irisstop 102 for light amount adjustment is placed. The photographingoptical system of the present invention is a zoom lens having a scalingability. Depending on the optical type, performing the scaling operationcauses a change in the size of the exit pupil or the distance from theimage plane. The photographing optical system shown in FIGS. 4A and 4Brepresents a state in which the focal length is set at the middlebetween the wide angle side and the telephoto side, that is, the Middlestate. The exit pupil distance in this state is represented by Zmid.Assuming that this distance is a standard exit pupil distance Znorm, theshape of the on-chip microlens is designed.

FIG. 4A is a view showing the conjugated relation between the firstpixel 211 and the photographing optical system. Note that the samereference numerals as in FIG. 1 denote the same components in FIG. 4A.Referring to FIG. 4A, a lens barrel member 101 b holds the first lensgroup 101, and a lens barrel member 105 b holds the third lens group105. An opening plate 102 a defines the aperture diameter in a fullaperture state, and diaphragm blades 102 b adjust the aperture diameterin a stopped-down-aperture state. Note that the members 101 b, 102 a,102 b, and 105 b which act to limit the light beam passing through thephotographing optical system are illustrated as optical virtual imageswhen observed from the image plane. A composite opening near the stop102 is defined as the exit pupil of the lens, and the distance from theimage plane is defined as Zmid, as described above.

Referring to FIG. 4A, the first pixel 211 includes the photoelectricconversion units 211 a to 211 d, wiring layers 211 e to 211 g, a colorfilter 211 h, and the on-chip microlens 211 i from the lowermost side.The photoelectric conversion units 211 a and 211 b overlap in thedirection perpendicular to the drawing surface (the Y-axis direction).The photoelectric conversion units 211 c and 211 d also overlap in asimilar manner. These photoelectric conversion units are projected ontothe exit pupil plane of the photographing optical system through theon-chip microlens 211 i as projected images EP1 a to EP1 d,respectively.

When the stop 102 is in the full aperture state (for example, F2.8), theoutermost portion of the light beam passing through the photographingoptical system is represented by L(F2.8). The pupil projected images EP1a to EP1 d are not vignetted by the stop opening. On the other hand,when the stop 102 is in the stopped-down-aperture state (for example,F5.6), the outermost portion of the light beam passing through thephotographing optical system is represented by L(F5.6). The outer sidesof the pupil projected images EP1 a to EP1 d are vignetted by the stopopening. However, at the center of the image plane, the vignetted statesof the projected images EP1 a to EP1 d are symmetrical about the opticalaxis, and the amounts of light received by the photoelectric conversionunits 211 a to 211 d are equal.

FIG. 4B is a view showing the conjugated relation between the secondpixel 222 and the exit pupil plane of the photographing optical system.FIG. 4B is different from FIG. 4A for explaining the conjugated relationof the first pixel 211 only in the shapes of the photoelectricconversion units of the pixel. That is, the projected images of thephotoelectric conversion units 222 a to 222 d are formed on the exitpupil plane of the photographing optical system as EP22 a to EP22 d.

As a result, in the full aperture state, sufficient light beams areincident on the four photoelectric conversion units 222 a to 222 d. Inthe stopped-down-aperture state, the light beams to the photoelectricconversion units 222 a and 222 d are almost shielded. However, lightbeams are incident on the photoelectric conversion units 222 b and 222 cto some extent. For this reason, focus detection can be performed evenin the stopped-down-aperture state.

FIGS. 5A to 5D show images formed by projecting the photoelectricconversion units of a pixel arranged at the center of the image planeonto the exit pupil plane of the photographing optical system.

FIG. 5A is a plan view showing the projected images of the photoelectricconversion units 211 a to 211 d of the first pixel 211. TL2(F2.8)represents the exit pupil in the full aperture state of thephotographing optical system, and TL2(F5.6) represents the exit pupil inthe stopped-down-aperture state. EP1 a to EP1 d are the projected imagesof the photoelectric conversion units 211 a to 211 d of the first pixel211. As described with reference to FIG. 4A, in the pixel arranged atthe center of the image plane, the connection center of the fourphotoelectric conversion units 211 a to 211 d matches the center of theexit pupil of the photographing optical system. Hence, the four pupilprojected images EP1 a to EP1 d are uniformly eclipsed from theperipheral portion in accordance with the aperture state of thephotographing optical system. In the stopped-down-aperture state, thelight reception amounts of the photoelectric conversion units 211 a to211 d decrease, and the decrease amounts are equal.

FIG. 5B shows the light reception characteristic on a section takenalong a line A-A in FIG. 5A. The abscissa represents the horizontalcoordinate on the exit pupil plane of the photographing optical system,and the ordinate represents the light reception efficiency of eachphotoelectric conversion unit. As described with reference to FIGS. 4Aand 4B, the photoelectric conversion units arranged in the pixel and theexit pupil of the photographing optical system have a conjugatedrelation through the on-chip microlens. This means that only a lightbeam passing through the common region of the exit pupil TL of thephotographing optical system on the exit pupil plane and the projectedimage EP of a photoelectric conversion unit reaches the photoelectricconversion unit. Hence, the projected image corresponds to the aperturestop unique to the pixel arranged on the exit pupil plane of thephotographing optical system, and the ordinate of FIG. 5B represents thetransmittance distribution of each aperture stop. The transmittancedistribution can be regarded as the light beam reception efficiency ofeach photoelectric conversion unit. The distribution characteristic ofthe light beam reception efficiency will be referred to as a “pupilintensity distribution” for the descriptive convenience.

If the projection performance of the on-chip microlens 211 i isstigmatic from the viewpoint of geometrical optics, the pupil intensitydistribution is represented by a step function having only one of values“0” and “1”. However, since the size of each pixel is as small asseveral μm, the sharpness of the image of a photoelectric conversionunit projected onto the exit pupil plane becomes lower due todiffraction of light. In addition, since the on-chip microlens 211 i isnormally a spherical lens, the sharpness of the projected image alsolowers due to spherical aberration. Hence, the pupil intensitydistributions of the pixels also become dull so as to have roundedshoulders and long tails on both sides, as shown in FIG. 5B.

The relationship between the pupil intensity distribution and the focusdetection characteristic will be described next. In a pair of pupilintensity distributions in the X-axis direction, the separation distancebetween the gravity centers of the portions extracted in the exit pupilrange of the photographing optical system corresponds to the base-linelength in the phase difference focus detection system. In this case, thebase-line length is defined by an angle θ (unit: radian) obtained bydividing the gravity center separation distance (unit: mm) on the pupilplane of the photographing optical system by the pupil distance (unit:mm). Letting u (unit: mm) be the horizontal shift amount of a pair ofimages at the time of focus detection, and DEF (unit: mm) be the defocusamount at that time, the relationship is represented byθ×DEF=u  (1)

The angle θ takes a different value for each f-number of thephotographing optical system. In FIG. 5B, the base-line lengths for F2.8and F5.6 are represented by θ(F2.8) and θ(F5.6), respectively. Accordingto equation (1), the larger the base-line length θ is, the larger thehorizontal shift amount of the focus detection image for the unitdefocus amount is, and the higher the focus detection accuracy is. Onthe other hand, in the large defocus state, the horizontal shift amountof the pair of images also increases. Hence, if the focus detection areais narrow, the maximum defocus amount that allows focus detectionundesirably decreases.

If the pupil intensity distribution largely spreads in the X direction,the light reception amount of each photoelectric conversion unitincreases, resulting in little noise when the signal is used as an imagesignal or an improved low luminance detection limit when the signal isused as a focus detection signal. On the other hand, the image blur inthe large defocus state also increases. This lowers the contrast of thefocus detection signal and undesirably decreases the maximum defocusamount that allows focus detection. The first pixel 211 is advantageousfor highly accurate focus detection when the defocus amount is smallbecause the spread width of the pupil intensity distribution is large,and the base-line length is large.

FIG. 5C is a plan view showing the projected states of the photoelectricconversion units 222 a to 222 d of the second pixel 222. The two circlesindicate the exit pupil of the photographing optical system in the fullaperture state and that in the stopped-down-aperture state as in FIG.5A. EP22 a to EP22 d are the projected images of the photoelectricconversion units 222 a to 222 d of the second pixel 222.

FIG. 5D shows pupil intensity distributions representing thecharacteristic on a section in FIG. 5C. In the first embodiment, theoutputs of the photoelectric conversion units 222 b and 222 c are usedat the time of focus detection. The pupil intensity distributions EP22 band EP22 c of the photoelectric conversion units are narrower than thoseof the first pixel shown in FIG. 5B. Hence, even when the defocus amountis large, the blur of AF images formed by the focus detection pixels iskept small, and focus detection never becomes impossible. In addition,the degree of vignetting is slight in the pupil intensity distributionsEP22 b and EP22 c even in the stopped-down-aperture state of thephotographing optical system. Hence, the change in the base-line lengthis small, that is, θ(F2.8)=θ(F5.6), and focus detection is possible evenin the stopped-down-aperture state. That is, in the second pixel 222,since the spread width of the pupil intensity distribution is small, andthe base-line length is small, focus detection is possible even in thestopped-down-aperture state or when the defocus amount is large.

As described above, the base-line length controls the focus detectionaccuracy and the maximum detection defocus amount, however thesecharacteristics have trade-off relationship. The width of the pupilintensity distribution controls the light reception amount and the imageblur, and these characteristics also have trade-off relationship. In thepresent invention, the first pixel 211 and the second pixel 222, whichhave different characteristics, are selectively used in accordance withthe focus detection condition, as will be described later, therebyimproving the focus detection capability.

FIGS. 6A and 6B are views showing the conjugated relation between thephotoelectric conversion units of a pixel at the peripheral image heightand the exit pupil plane of the photographing optical system when thezoom state is the Middle state. The individual light beam shieldingmembers at the peripheral image height portion are almost equal to thoseat the center of the image plane. However, since a plurality of virtualopenings having different distances are viewed from an obliquedirection, the exit pupil shape as the composite opening changes tocause so-called vignetting. For example, in the full aperture state, theouter light beam L(F2.8) called a lower line is regulated by the lensbarrel member 101 b, and the outer light beam L(F2.8) called an upperline is regulated by the lens barrel member 105 b. Hence, the size ofthe composite opening in the X-direction on the exit pupil plane of apixel at the peripheral image height is smaller than the opening size ofa pixel at the center image height. On the other hand, in thestopped-down-aperture state, the opening portion of the diaphragm blades102 b serves as the exit pupil without any influence of vignetting.

The arrangement of the image sensor at the peripheral image height willbe described next. A pixel arranged at a position with a large imageheight obliquely receives the light beam from the exit pupil. Hence, theon-chip microlens needs to decenter toward the image plane center withrespect to the connection center of the photoelectric conversion regionsof the pixel. The optimum decentering amount depends on the image heightand the distance to the exit pupil of the photographing optical system.For the dependence of the decentering amount on the image height, adecentering amount proportional to the image height is generally given.On the other hand, since the distance of the exit pupil changesdepending on the zoom state or focus state of the photographing opticalsystem, it is necessary to uniquely determine a representative state. Inthe first embodiment, the representative state of the zoom state is theMiddle state, and the representative state of the focus state is thein-focus state for an infinite object. The exit pupil distance at thisstate is defined as Znorm.

FIG. 6A shows the projection relationship of the first pixel 211. Theconnection center of the four pupil projected images EP1 a to EP1 d isprojected onto the exit pupil plane at a distance Zmid (=Znorm) from theimage plane without decentering with respect to the center of the exitpupil.

As a result, in the full aperture state, the light reception amount ofthe pixel decreases due to vignetting, and the decrease amount is almostthe same between the four photoelectric conversion units. In thestopped-down-aperture state, the light amount is almost the same as thatof the pixel arranged at the center of the image plane shown in FIG. 4A.

FIG. 6B shows the conjugated relation between the second pixel 222 andthe exit pupil plane of the photographing optical system. The vignettingof the photographing optical system and the decentering state of theon-chip microlens of the pixel are the same as in FIG. 6A. Hence, theprojected images of the photoelectric conversion units 222 a to 222 dare formed on the exit pupil plane at the distance Zmid from the imageplane as EP22 a to EP22 d.

As a result, in the full aperture state as well, most of the light beamsto the outer photoelectric conversion units 222 a and 222 d are shieldedby vignetting. In the stopped-down-aperture state, the light beams arealmost completely shielded. On the other hand, since light beams enterthe photoelectric conversion units 222 b and 222 c to some extent inboth the full aperture state and the stopped-down-aperture state, focusdetection is possible even in the stopped-down-aperture state.

FIGS. 7A to 7D show the projected images of the photoelectric conversionunits of the pixel arranged at the peripheral image height portion onthe exit pupil plane of the photographing optical system.

FIG. 7A is a plan view showing the projected images of the photoelectricconversion units 211 a to 211 d of the first pixel 211. TL2(F2.8)represents the exit pupil in the full aperture state of thephotographing optical system. The exit pupil has a shape formed bycombining a plurality of arcs because of the effects of vignettingdescribed with reference to FIGS. 6A and 6B. TL2(F5.6) represents theexit pupil in the stopped-down-aperture state. The exit pupil has acircular opening without any influence of vignetting. EP1 a to EP1 d arethe projected images of the photoelectric conversion units 211 a to 211d of the first pixel 211. In the pixel at the peripheral image height,the pupil projected images EP1 a to EP1 d are vignetted not only in thestopped-down-aperture state but also in the full aperture state. Theconnection center of the four photoelectric conversion units 211 a to211 d matches the center of the exit pupil of the photographing opticalsystem. Hence, the pupil projected images EP1 a to EP1 d are vignettedsymmetrically about the axis. For this reason, the light receptionamounts of the photoelectric conversion units 211 a to 211 d decrease,and the decrease amounts are equal.

FIG. 7B is a graph for explaining the pupil intensity distributions inFIG. 7A. The pupil intensity distributions of the pupil projected imagesEP1 a to EP1 d are the same as those in FIG. 5B. On the other hand,since the exit pupil width of the photographing optical system in thefull aperture state becomes smaller, the base-line length θ(F2.8) isshorter than in FIG. 5B.

FIG. 7C is a plan view showing the projected images of the photoelectricconversion units 222 a to 222 d of the second pixel 222. FIG. 7D showsthe pupil intensity distributions of the projected images. In FIG. 7D aswell, the pupil intensity distributions EP22 a to EP22 d of thephotoelectric conversion units 222 a to 222 d and their pupil intensitydistributions are almost the same as those in FIG. 5D. In addition, theexit pupil width of the photographing optical system is the same as inFIG. 7A. Hence, the pupil intensity distributions EP22 b and EP22 c forfocus detection are not vignetted by the exit pupil in the full aperturestate and the stopped-down-aperture state. Hence, the base-line lengthsmaintain the relation θ(F2.8)=θ(F5.6), and focus detection is possibleeven in the stopped-down-aperture state.

FIGS. 8A and 8B are views showing the conjugated relation between thephotoelectric conversion units of a pixel at the peripheral image heightand the exit pupil plane of the photographing optical system when thezoom state is Wide (wide angle side). In the photographing opticalsystem of the first embodiment, the exit pupil distance from the imageplane changes depending on the zoom state. In the Wide state, thedistance between the image plane and the exit pupil is shorter than theabove-described standard distance Znorm. On the other hand, as describedwith reference to FIGS. 6A and 6B, the decentering amount of the on-chipmicrolens is optimized based on the exit pupil distance, that is, Znormwhen the zoom state is Middle. In the Wide state, the decentering amountof the on-chip microlens does not have the optimum value. The connectioncenter of the projected images of the photoelectric conversion units ofthe pixel at the peripheral image height decenters with respect to thecenter of the exit pupil of the photographing optical system.

FIG. 8A shows the projection relationship of the first pixel 211. Theprojected images of the four photoelectric conversion units 211 a to 211d are formed on the exit pupil plane at a distance Zwide from the imageplane as EP1 a to EP1 d. The connection center of the four pupilprojected images EP1 a to EP1 d decenters in the negative direction onthe X-axis with respect to the center of the exit pupil of thephotographing optical system. As a result, in the full aperture state,the light reception amount of the pixel decreases due to vignetting, andthe decrease amount is nonuniform between the four photoelectricconversion units 211 a to 211 d. The nonuniformity of the lightreception amount becomes more conspicuous as the stop aperture diameterdecreases.

FIG. 8B is a view showing the conjugated relation between the secondpixel 221 and the exit pupil plane of the photographing optical system.When the zoom state of the photographing optical system is Middle, thesecond pixel 222 is used, as shown in FIG. 6B. When the zoom state isWide, the second pixel 221 is used. As for the photoelectric conversionunits 221 a to 221 d of the second pixel 221, the connection center ofthe photoelectric conversion units 221 b and 221 c is shifted in thenegative direction on the X-axis, as shown in FIGS. 2 and 8B. On theother hand, the exit pupil distance Zwide of the photographing opticalsystem is shorter than the standard distance Znorm. Hence, theconnection center of the photoelectric conversion units 221 b and 221 cis projected on the exit pupil plane without decentering.

FIGS. 9A and 9B are views showing the conjugated relation between thephotoelectric conversion units of a pixel at the peripheral image heightand the exit pupil plane of the photographing optical system when thezoom state is Tele (telephoto side). In the Tele state, the distancebetween the image plane and the exit pupil is longer than the standarddistance Znorm, contrary to the Wide state shown in FIGS. 8A and 8B.Hence, the connection center of the projected images of thephotoelectric conversion units of the pixel at the peripheral imageheight decenters with respect to the center of the exit pupil of thephotographing optical system, and the decentering direction is reverseto that in the Wide state.

FIG. 9A shows the projection relationship of the first pixel 211. Theprojected images of the four photoelectric conversion units 211 a to 211d are formed on the exit pupil plane at a distance Ztele from the imageplane as EP1 a to EP1 d. The connection center of the four pupilprojected images EP1 a to EP1 d decenters in the positive direction onthe X-axis with respect to the center of the exit pupil of thephotographing optical system. As a result, in the full aperture state,the light reception amount of the pixel decreases due to vignetting, andthe decrease amount is nonuniform between the four photoelectricconversion units 211 a to 211 d. The nonuniformity of the lightreception amount becomes more conspicuous as the stop aperture diameterdecreases.

FIG. 9B is a view showing the conjugated relation between the secondpixel 223 and the exit pupil plane of the photographing optical system.When the zoom state of the photographing optical system is Tele, thesecond pixel 223 is used. As for the photoelectric conversion units 223a to 223 d of the second pixel 223, the connection center of thephotoelectric conversion units 223 b and 223 c is shifted in thepositive direction on the X-axis, as shown in FIGS. 2 and 9B. On theother hand, the exit pupil distance Ztele of the photographing opticalsystem is longer than the standard distance Znorm. Hence, the connectioncenter of the photoelectric conversion units 223 b and 223 c isprojected on the exit pupil plane without decentering.

FIGS. 10A to 13 are views for explaining the projection positionrelationship between the photoelectric conversion units of the pixelsand the exit pupil of the photographing optical system in five focusdetection areas on the image sensor 107.

FIG. 10A is a view of the image sensor 107 viewed from the photographingoptical system side. In the image sensor of the present invention, thefocus detection pixels are discretely arranged all over the imagingarea, as described with reference to FIG. 2. For this reason, focusdetection is possible at an arbitrary position. However, since the exitpupil distance of the photographing optical system changes depending onthe zoom state, the conjugated relation between the photoelectricconversion units and the exit pupil changes depending on the position(image height) on the image sensor, as described with reference to FIGS.4A to 9B. The projection relationship will be described here byexemplifying five points on the image sensor 107, as shown in FIG. 10A.AFW1 to AFW5 indicate the representative positions of the center, upper,lower, left, and right focus detection areas, respectively.

FIG. 10B shows only the photoelectric conversion units of the firstpixels 211, the second pixels 221 to 223, and the third pixels 224 to226 extracted in the five focus detection areas AFW1 to AFW5 shown inFIG. 10A. Solid rectangles indicate the photoelectric conversion unitsto be used for focus detection. Not all the photoelectric conversionunits are used simultaneously, and they are selectively used inaccordance with the zoom state of the photographing optical system, aswill be described later.

FIG. 11 shows the projected images of the exit pupil on thephotoelectric conversion units when the zoom state of the photographingoptical system is Middle. FIGS. 5A and 5B and FIGS. 7A and 7B show theprojected images of the photoelectric conversion units on the exit pupilplane of the photographing optical system. To the contrary, FIG. 11shows the projected images of the exit pupil of the photographingoptical system on the uppermost surfaces of the photoelectric conversionunits. These drawings actually explain the same situation because theexit pupil and the photoelectric conversion units have the conjugatedrelation through the on-chip microlenses. FIGS. 5A to 5D and FIGS. 7A to7D show two f-numbers of the photographing optical system, that is, F2.8(full aperture state) and F5.6 (stopped-down-aperture state). FIG. 11shows only the exit pupil when the f-number is F5.6.

Referring to FIG. 11, the exit pupil distance Zmid of the photographingoptical system equals the standard distance Znorm. Hence, independentlyof the position on the image sensor 107, the connection centers of thephotoelectric conversion units 211 a to 211 d of the first pixels 211match the pupil projected images EP1 a to EP1 d of the exit pupilwithout being decentering. For focus detection, out of the second pixels221 to 223 and the third pixels 224 to 226, the second pixels 222 andthe third pixels 225 in which the connection centers of the focusdetection photoelectric conversion units are not shifted are selectedindependently of the position on the image sensor 107.

FIG. 12 shows the projected images of the exit pupil on thephotoelectric conversion units when the zoom state of the photographingoptical system is Wide. In the Wide state, the exit pupil distance Zwideof the photographing optical system is shorter than the standarddistance Znorm. Hence, the centers of the projected images of the exitpupil on the photoelectric conversion units decenter isotropicallyoutward from the central position of the image sensor, and thedecentering amount is proportional to the image height. In, for example,the focus detection area AFW2, the exit pupil decenters in the positivedirection of the Y-axis. For this reason, out of the second pixels 221to 223, the second pixel 222 in which the connection center of the focusdetection photoelectric conversion units is not shifted is selected. Inaddition, out of the third pixels 224 to 226, the third pixel 224 inwhich the connection center of the focus detection photoelectricconversion units is shifted in the positive direction of the Y-axis isselected. In the focus detection area AFW4, the exit pupil decenters inthe negative direction of the X-axis. For this reason, out of the secondpixels 221 to 223, the second pixel 221 in which the connection centerof the focus detection photoelectric conversion units is shifted in thenegative direction of the X-axis is selected. In addition, out of thethird pixels 224 to 226, the third pixel 225 in which the connectioncenter of the focus detection photoelectric conversion units is notshifted in the Y-axis direction is selected.

It should be noted here that the pixels selected from the second pixels221 to 223 and the third pixels 224 to 226 change depending on theposition of the focus detection area. That is, the different kinds ofsecond pixels 221, 222, and 223 are selected as the optimum secondpixels in the focus detection areas AFW4, AFW1, and AFW5 having the sameY-coordinate, respectively. In addition, the different kinds of thirdpixels 224, 225, and 226 are selected as the optimum third pixels in thefocus detection areas AFW2, AFW1, and AFW3 having the same X-coordinate,respectively.

FIG. 13 shows the projected images of the exit pupil on thephotoelectric conversion units when the zoom state of the photographingoptical system is Tele. In the Tele state, the exit pupil distance Zteleof the photographing optical system is longer than the standard distanceZnorm. Hence, the centers of the projected images of the exit pupil onthe photoelectric conversion units decenter isotropically inward fromthe central position of the image sensor, and the decentering amount isproportional to the image height. In, for example, the focus detectionarea AFW2, the exit pupil decenters in the negative direction of theY-axis. For this reason, out of the second pixels 221 to 223, the secondpixel 222 in which the connection center of the focus detectionphotoelectric conversion units is not shifted is selected. In addition,out of the third pixels 224 to 226, the third pixel 226 in which theconnection center of the focus detection photoelectric conversion unitsis shifted in the negative direction of the Y-axis is selected. In thefocus detection area AFW4, the exit pupil decenters in the positivedirection of the X-axis. For this reason, out of the second pixels 221to 223, the second pixel 223 in which the connection center of the focusdetection photoelectric conversion units is shifted in the positivedirection of the X-axis is selected. In addition, out of the thirdpixels 224 to 226, the third pixel 225 in which the connection center ofthe focus detection photoelectric conversion units is not shifted in theY-axis direction is selected.

Although the selected focus detection pixels change depending on theposition of the focus detection area, as in FIG. 12, the direction isreversed. That is, the different kinds of second pixels 223, 222, and221 are selected as the optimum second pixels in the focus detectionareas AFW4, AFW1, and AFW5 having the same Y-coordinate, respectively.In addition, the different kinds of third pixels 226, 225, and 224 areselected as the optimum third pixels in the focus detection areas AFW2,AFW1, and AFW3 having the same X-coordinate, respectively.

FIGS. 14A to 14C show the output waveforms of the first pixels 211 andthe second pixels 222 at the time of focus detection. The abscissarepresents the X-coordinate of the pixel, and the ordinate representsthe pixel signal output. The zoom state is Middle. The f-number is thefull aperture state (F2.8). The focus detection area is AFW1 at thecenter of the image plane. The object has a vertical-striped patternhaving a luminance change in the X-axis direction but no luminancechange in the Y-axis direction. In this case, the second pixels 222 thatdivide the pupil in the X-axis direction is used for focus detection. Inthe second pixels 222, the outputs of the photoelectric conversion units222 b and 222 c at the center are used for focus detection, as shown inFIG. 2 or 11. The output waveforms are indicated by AFb and AFc in FIGS.14A to 14C. In the first pixels 211, the sum signal of the photoelectricconversion units 211 a and 211 b juxtaposed in the Y-axis direction andthe sum signal of the photoelectric conversion units 211 c and 211 d areusable as a pair of focus detection signals. The pair of image signalsare indicated by IMab and IMcd.

FIG. 14A shows the waveforms when the defocus amount is 0, that is, inthe in-focus state. In the in-focus state, all the signals IMab and IMcdof the first pixels 211 and the signals AFb and AFc of the second pixels222 are in phase in the X direction without any lateral shift. Thesignal strength is proportional to the light reception amount of eachpixel. As described with reference to FIGS. 5A and 5C, the lightreception amounts of the second pixels 222 is smaller than those of thefirst pixels 211 when the f-number is F2.8. Hence, in FIG. 14A, thesignal strengths hold a relationship given byIMab=IMcd>AFb=AFc  (2)

FIG. 14B shows the waveforms when the focus shift amount is relativelysmall, for example, when the defocus amount is 2 mm. The pair of imagesignals IMab and IMcd obtained from the first pixel 211 generates animage shift amount u1. The pair of image signals AFb and AFc obtainedfrom the second pixel 222 generates an image shift amount u2. Since thebase-line length of each of the first pixels 211 is larger, therelationship between the image shift amounts is represented by u1>u2. Onthe other hand, when the defocus amount is small, the decrease in thecontrast caused by the blur of each focus detection image is slight (thecontrast is equal to or higher than a predetermined threshold), and allimage signals hold sufficient contrast information. Hence, when thedefocus amount is small, as shown in FIG. 14B, focus detection ispossible by both the first pixels 211 and the second pixels 222.However, since each of the first pixels 211 has a larger base-linelength and a higher focus detection accuracy, the in-focus state ispreferably controlled preferentially using the image shift amount u1.

FIG. 14C shows the waveforms when the focus shift amount is relativelylarge, for example, when the defocus amount is 10 mm. In this case aswell, the pair of image signals IMab and IMcd obtained from the firstpixels 211 generates the image shift amount u1. The pair of imagesignals AFb and AFc obtained from the second pixels 222 generates theimage shift amount u2. The relationship between them is represented byu1>u2. However, since the pupil intensity distribution of the firstpixels 211 largely spreads, the image blur is large, and the contrast ofthe focus detection signals greatly lowers (that is, the contrast islower than the predetermined threshold). Hence, the reliability ofcorrelation for detecting the lateral shift amount of the pair of imagesalso lowers, and the possibility of detection errors increases. On theother hand, since the second pixels 222 have a sharp pupil intensitydistribution, the image blur is small, and the contrast of the focusdetection signals maintains a relatively high value. For this reason,when the defocus amount is large, the in-focus state is preferablycontrolled preferentially using the image shift amount u2 of the secondpixels 222.

A case in which the focus detection area is located at the center hasbeen described with reference to FIGS. 14A to 14C. When the zoom stateis Middle, the projection relationship between the photoelectricconversion units of the pixels and the exit pupil of the photographingoptical system have no image height dependence. Hence, in the focusdetection areas at the periphery of the screen, basically, the samecharacteristics as in FIGS. 14A to 14C can be obtained, although thestrengths of the output waveforms IMab and IMcd of the first pixels 211slightly lower due to the decrease in the light amount caused byvignetting of the photographing optical system.

FIGS. 15A and 15B show the output waveforms when the conjugated relationbetween the photoelectric conversion units of a pixel and the exit pupilof the photographing optical system changes depending on the imageheight. The zoom state is Wide. The f-number is thestopped-down-aperture state (F.5.6). The focus state is in-focus (thedefocus amount is 0). As the focus detection areas, AFW1 at the centerand AFW4 at the periphery are shown.

FIG. 15A shows the output waveforms in the focus detection area AFW1 atthe center, which correspond to the signals output from the first pixels211 and the second pixels 222 in the focus detection area AFW1 shown inFIG. 12. In this case, the signals IMab and IMcd obtained from the firstpixels 211 match. The signals AFb and AFc obtained from the secondpixels 222 also match. The relationship between the signal strengths isdetermined by the shape of each photoelectric conversion unit and thearea of the projected image of the exit pupil corresponding to F5.6 onthe photoelectric conversion unit. In the first embodiment, therelationship is given byIMab=IMcd≧AFb=AFc  (3)Hence, in this state, almost the same focus detection accuracy can beobtained using either of the signals of the first and second pixels.

FIG. 15B shows the output waveforms in the focus detection area AFW4 atthe periphery, which correspond to the signals output from the firstpixels 211 and the second pixels 221 in the focus detection area AFW4shown in FIG. 12. In this case, since the second pixels 221 in which thedecentering of the projected image of the exit pupil of thephotographing optical system is minimum is selected, the outputs AFb andAFc match. On the other hand, in the first pixels 211, the decenteringof the exit pupil cannot be canceled, and the pair of signals IMab andIMcd has a large difference. Hence, a relationship is as follows.IMab>AFb=AFc>IMcd  (4)That is, since the strength of the signal IMcd, which is one of thesignals obtained from the first pixels 211, largely lowers, thereliability of focus detection calculation using the signal is low.Hence, in this state, focus detection is preferably performed using thesignals of the second pixels 221.

FIGS. 16A to 16C conceptually illustrate a method of adding the outputsof the photoelectric conversion units and a pixel interpolation methodwhen creating a 3D image. In the first embodiment, each pixel has aplurality of photoelectric conversion units all over the image sensor107 so that a parallax image can be obtained. Hence, a 3D image can becreated by the following method.

FIG. 16A shows the arrangements of three kinds of pixels and illustratesthe shapes of the photoelectric conversion units of the first pixel 211,the second pixel 222, and the third pixel 225 from the left.

FIG. 16B is a view for explaining a pixel signal processing method whenthe user holds the camera in the landscape orientation, that is, whenthe user holds the camera to make the negative direction of the Y-axisof the image sensor 107 shown in FIG. 2 match the direction of gravity.The human eyes are arranged along the horizontal axis orthogonal to thedirection of gravity. For this reason, to create a 3D image, theextended line of the base line for generating a parallax is preferablyparallel to the horizontal axis. Hence, when the attitude detection unit116 shown in FIG. 1 detects the attitude of the camera, and it isconsequently determined that the direction of gravity is directeddownward in FIG. 16B, the signals of the photoelectric conversion unitsare processed in the following way.

For the first pixel 211, the sum of the signals of the photoelectricconversion units 211 a and 211 b serves as one signal of the parallaximage, and the sum of the signals of the photoelectric conversion units211 c and 211 d serves as the other signal. For the second pixel 222,the sum of the signals of the photoelectric conversion units 222 a and222 b serves as one signal of the parallax image, and the sum of thesignals of the photoelectric conversion units 222 c and 222 d serves asthe other signal. This operation allows to obtain a 3D image signalequivalent to the first pixel 211. On the other hand, in the othersecond pixels 221 and 223, the segmented shape of the photoelectricconversion units is asymmetrical in the X direction. It is thereforeimpossible to obtain a 3D image signal equivalent to the first pixel 211by performing the same addition as in the second pixel 222. For thesecond pixels 221 and 223, a 3D image signal is created by the sameinterpolation calculation as in the third pixels to be described next.

The third pixel 225 has no parallax information in the horizontal axisdirection. Hence, a pair of parallax signals is created by interpolationfrom the four first pixels 211 adjacent in diagonal directions at anangle of 45°. This also applies to the other third pixels 224 and 226.With the above-described processing, a pair of signals can be obtainedin every pixel. Note that all the above-described processes are executedby the CPU 121 of the camera.

FIG. 16C is a view for explaining a pixel signal processing method whenthe user holds the camera in the portrait orientation, that is, when theuser holds the camera to make the negative direction of the X-axis ofthe image sensor 107 shown in FIG. 2 match the direction of gravity. Inthis case, the addition processing of the photoelectric conversion unitsis performed in a direction orthogonal to that in FIG. 16B. For thefirst pixel 211, the sum of the signals of the photoelectric conversionunits 211 a and 211 c serves as one signal of the parallax image, andthe sum of the signals of the photoelectric conversion units 211 b and211 d serves as the other signal. The second pixels have no parallaxinformation in the horizontal axis direction, that is, in the verticaldirection of FIG. 16C. Hence, a pair of parallax signals is created byinterpolation from the four first pixels 211 adjacent in diagonaldirections at an angle of 45°. For the third pixel 225, the sum of thesignals of the photoelectric conversion units 225 a and 225 b serves asone signal of the parallax image, and the sum of the signals of thephotoelectric conversion units 225 c and 225 d serves as the othersignal.

With the above-described processing, the direction of gravity acting onthe camera is detected, and a 3D image is created based on the result.Note that the user can select in advance whether to switch thecombination of signals to be added in accordance with the direction ofgravity, and this will be described later with reference to flowcharts.

FIGS. 17A and 17B are views for explaining an image and focus detectionsignals acquired at the time of focus detection and a defocus mapobtained from the focus detection result. Referring to FIG. 17A, theobject image formed in the imaging plane includes a person at a closedistance in the middle, a tree at an intermediate distance on the leftside, and a mountain at a far distance on the upper right side. A casewill be explained in which the signals of the second pixels 222 areemployed as the focus detection signals of the image shown in FIG. 17A.

In FIG. 17A, the face of the person exists at the center of the screen.When the presence of the face is detected by a known face recognitiontechnique, the pair of focus detection signals AFb and AFc of the secondpixels 222 are obtained about the face region. For regions other thanthe face region, focus detection areas are set all over thephotographing screen at a predetermined pitch. A focus detection areacorresponding to the tree trunk and the signals thereof are shown on theleft side of FIG. 17A. A focus detection area corresponding to themountain ridgeline and the signals thereof are shown on the right sideof FIG. 17A. Since a pair of signals obtained in each focus detectionarea is laterally shifted, a lateral shift amount u is calculated byknown correlation, and the defocus amount is calculated using equation(1).

After that, for the main object, that is, the face region located at thecenter in FIGS. 17A and 17B, the focus lens of the photographing opticalsystem is driven so that the defocus amount becomes 0, and the focusdetection is performed again.

With the above-described focus adjustment process, focus shiftinformation, that is, a so-called defocus map in the entirephotographing screen can be acquired. An example is shown in FIG. 17B.FIG. 17B illustrates an example in which the defocus amounts areintegrated, based on a predetermined resolution, into DEF0 to DEF3sequentially from the region with a small defocus amount.

FIGS. 18 to 20 are flowcharts for explaining focus adjustment processingand photographing processing of the camera according to the firstembodiment of the present invention. The processing will be describedbelow also with reference to FIGS. 1 to 17B described above.

FIG. 18 is a flowchart showing the procedure of photographing processingaccording to the first embodiment. In step S101, the user turns on thepower switch of the camera. In step S102, the CPU 121 checks theoperations of the actuators and the image sensor 107 in the camera,initializes the memory contents and execution programs, and executes aprephotographing operation.

In step S103, the CPU receives photographing condition settings. Morespecifically, the CPU 121 receives the exposure adjustment mode, thefocus adjustment mode, the image mode (2D or 3D), the image quality (thenumber of recording pixels, compression ratio, and the like), and thelike set by the user.

In step S104, it is determined whether the 3D recording mode is set. Ifthe 3D recording mode is set, the CPU 121 fixes the f-number at the timeof photographing to the full aperture state in step S105. For the 3Drecording, a pair of images needs to have appropriate parallaxinformation, and the parallax information decreases when the stop of thephotographing optical system is set in the stopped-down-aperture stateto adjust the light amount. Hence, in the 3D recording mode, the stop isfixed in the full aperture state, and the exposure amount is adjusted bythe accumulation time of the image sensor 107. Upon determining in stepS104 that the 2D mode is set, the CPU 121 controls the f-number to adesignated value in step S106. The designated value here is an f-numberselected by the user in aperture priority AE or a preset f-number basedon the exposure control program in program AE.

In step S107, the zoom state, focus lens state, and stop state of thephotographing optical system are detected, and pieces of informationsuch as the size of the exit pupil and the exit pupil distance are readout from the ROM. In step S108, the image sensor 107 starts the imagecapturing operation and reads pixel signals. In step S109, a reducedimage for display is created from the read pixel signals and displayedon the display device 131 provided on the rear surface of the camera.The user can determine the composition or perform the zoom operationwhile visually checking the preview image.

In step S131, a focus detection subroutine to be described later isexecuted. In step S151, the CPU 121 determines whether the focus lensdriving amount calculated in step S131 is equal to or smaller than apredetermined value. If the focus lens driving amount is equal to orsmaller than the predetermined value, the CPU 121 determines that thein-focus state is obtained, and the process advances to step S153. Ifthe focus lens driving amount exceeds the predetermined value, the focuslens is driven in step S152.

In step S153, the CPU 121 determines whether the photographing switch ison. If the switch is not on, the process advances to step S181. If theswitch is on, the CPU executes, in step S161, an image recordingsubroutine to be described later.

In step S181, the CPU 121 determines the state of the main switch. Ifthe on state is maintained, the process returns to step S102 torepetitively execute the processing of steps S102 to S161 describedabove. Upon determining in step S181 that the main switch is off,processing from step S182 is executed.

In step S182, the image recorded in step S161 is transmitted to theserver computer via an Internet connection. Then, the server computerexecutes processing of large calculation scale such as reconstruction ofthe parallax information of the 3D image and accurate defocus mapcalculation. In step S183, the image processed by the server computer isreceived. In step S184, a corrected portion processed by the servercomputer is added or replacement correction is performed for theoriginal image recorded in step S161. In step S185, the photographingends.

FIG. 19 is a flowchart of the focus detection subroutine to be performedin step S131 of FIG. 18. In step S132, the object pattern is recognizedfrom the preview image, and face image determination, contrast analysisof the entire photographing screen, and the like are performed. In stepS133, the main object to be focused is determined based on therecognition result in step S132. In step S134, the exit pupil of thephotographing optical system is calculated based on the lens informationacquired in step S107 of FIG. 18. More specifically, the size of theexit pupil and its distance from the image plane are calculated, andvignetting for each image height is calculated. In step S135, pixelswhich are less affected by the vignetting and to be used for focusdetection are selected in each focus detection area based on the exitpupil information calculated in step S134. In step S136, a pair ofimages to be used for correlation is created from the outputs of thephotoelectric conversion units of each selected pixel. Note that onlyone type of pixels is not necessarily selected in step S136, and aplurality of types of pixels are selected if they are less affected byvignetting.

In step S137, so-called shading correction is performed for the createdfocus detection signals to reduce the unbalance of the light amountscaused by vignetting. This allows to reduce the strength differencebetween two images and improve the focus detection accuracy. In stepS138, correlation is performed to calculate the lateral shift amount uof the two images that have undergone the shading correction. In stepS139, the reliability of the image shift detection result is determinedbased on the level of matching between the two images calculated in thecorrelation process of step S138. A value with a low reliability is notemployed.

In step S140, the defocus amount is calculated using equation (1) fromthe reliable image shift amount u obtained in steps S138 and S139 andthe base-line lengths θ of the pixels used for focus detection. In stepS141, the defocus map in the entire photographing region is created.Note that the resolution (in the planar direction and the depthdirection) of the defocus map is set to such a value that does notaffect the recording rate of a moving image because the higher theresolution is, the longer the calculation time is. If a detailed defocusmap is necessary, the calculation is done in a high-performance servercomputer, as described concerning step S182 of FIG. 18. In step S142,the focus lens driving amount is calculated based on the main objectregion determined in step S133 and the defocus map created in step S141.In step S143, the process returns to the main routine.

FIG. 20 is a flowchart of the image recording subroutine to be performedin step S161 of FIG. 18. When the photographing switch is turned on, theattitude of the camera is detected in step S162. In step S163, additionof the photoelectric conversion units and pixel interpolation processingare performed based on the attitude detection result using the methodsdescribed with reference to FIGS. 16A to 16C. In step S164, a 3D imagecomplying with a predetermined format is created. In step S165, a normal2D image is created by erasing the parallax information from the imagecreated in step S164. The 2D image without the parallax information canbe obtained by, for example, adding the pixel information at the samecoordinates in a pair of images. In step S166, predetermined compressionprocessing is performed for the images created in steps S164 and S165,and the images are recorded in the flash memory 133.

In step S167, the defocus map created in step S141 of FIG. 19 isrecorded in association with the images. In step S168, the processreturns to the main routine.

As described above, according to the first embodiment, the first pixelincludes 2×2 photoelectric conversion units arranged in the X and Ydirections. The second pixel includes 4×1 photoelectric conversion unitsarranged only in the X direction. The third pixel includes 1×4photoelectric conversion units arranged only in the Y direction. At thetime of focus adjustment, the signals from the first pixels are usedunder the condition that the allowable value (in-focus accuracystandard) of focus detection errors is small, and highly accuratedistance measurement is necessary. The signals from the second pixelsand the third pixels are used under the condition that focus detectionis difficult to perform using the signals from the first pixels, forexample, when the exit pupil distance of the photographing opticalsystem is not appropriate, or the defocus amount is large. Hence,selectively using the first pixels, the second pixels, and the thirdpixels in accordance with the condition allows to reduce the probabilitythat focus detection is impossible and acquire a high-quality in-focusimage.

The total number of photoelectric conversion units is four in all thefirst pixel, the second pixel, and the third pixel. Hence, the pixelsare structurally different only in the shapes of the photoelectricconversion units and those of local electrodes for deriving electriccharges from there and can have the same structure except theseportions. Hence, the first pixel, the second pixel, and the third pixelhave almost the same electrical characteristics. Since the electricalcharacteristics can be substantially equal, it is possible to eliminatethe sensitivity unevenness or the like between the pixels and obtain ahigh-quality image. In addition, the circuit pattern of the image sensoris easy to design.

Upon focus detection, when all photoelectric conversion units of thefirst pixel are used, the base-line length increases, and highlyaccurate focus detection can be performed. This contributes to anincrease in in-focus accuracy. When the photoelectric conversion unitsof the second pixel and/or the third pixel are partially used, the blurand lateral shift of the focus detection image can be suppressed. Thiscontributes to widening of the detection limit in the large defocusstate. It is therefore possible to prevent the in-focus position frombeing lost and increase the in-focus accuracy even when the focus shiftamount is large.

The second pixel and the third pixel are especially useful in a largeout-of-focus state. However, a largely blurred scene out of aphotographed scene corresponds to a transient state, and the in-focusstate or almost in-focus state is obtained in most of the photographingtime. Therefore, by using a high density array of the first pixels, thefocus detection accuracy and stability in the in-focus state or almostin-focus state that takes up most of the photographed scene can beincreased, and a high-quality image can be obtained. The image capturingapparatus according to the present invention can acquire a 3D image. Toobtain a sufficient parallax in the 3D photographing mode, thephotographing optical system having a small f-number is often used nearthe full aperture state. In this case as well, the first pixel isuseful. Hence, a high density array of the first pixels is used, therebyincreasing the in-focus accuracy for the main object and obtaining ahigh definition 3D image. The density of the first pixels is preferablyhigher even for creation of the defocus map. The resolution of thedefocus map can be divided into the resolution in the planar directionand that in the depth direction. In an object area almost in focus, theresolutions can be high in both the planar and depth directions. In anarea largely out of focus, the resolutions can be low in bothdirections. Hence, when the arrangement density of the first pixels israised, a defocus map that satisfies the above-described characteristicsand has a well-balanced data amount and information accuracy can beobtained.

Second Embodiment

In the above-described first embodiment, the first pixel 211 includestotal of four photoelectric conversion units 211 a to 211 d segmentedinto two parts (integer m1=2) in the X direction and two parts (integern1=2) in the Y direction as well. Each of the second pixels 221 to 223includes total of four independent photoelectric conversion unitssegmented into four parts (integer m2=4) in the X direction but notsegmented (integer n2=1) in the Y direction.

In the second embodiment to be described below, the number of segmentsin the X direction is increased in all of a first pixel 211 and secondpixels 221 to 223. The pixels included in an image sensor 107 accordingto the second embodiment will be described below with reference to FIGS.21 and 22A to 22C. Note that since the arrangement is actually the sameas in the first embodiment except the number of photoelectric conversionunits, only the different points will be described, and a description ofsame or similar parts will be omitted.

FIG. 21 shows the pixel array of the image sensor 107 according to thesecond embodiment of the present invention. Note that a description ofthe same ability as in FIG. 2 will be omitted.

A first pixel 311 includes a total of six photoelectric conversion units311 a to 311 f segmented into three parts (integer m1=3) in the Xdirection and two parts (integer n1=2) in the Y direction. The sixphotoelectric conversion units 311 a to 311 f are divided so as to beline symmetry about the X- and Y-axes passing through the pixel center.That is, each of the divided photoelectric conversion units has arectangular planar shape long in the Y-axis direction. The connectedshape of the six regions is square. The photoelectric conversion unitshave the same segmented shape at all positions on the image plane. Theoutputs of the first pixel 311 are used for recording image generationand focus detection in a near in-focus state, like those of the firstpixel 211 of the above-described first embodiment. Note that the otherpixels having the same structure and arranged in the image sensor 107will also be referred to as the first pixels 311 hereinafter.

Second pixels 321 are discretely arranged among the first pixels 311 inaccordance with a predetermined array rule. The second pixel 321includes a total of six photoelectric conversion units 321 a to 321 fsegmented into six parts (integer m2=6) in the X direction but notsegmented (integer n2=1) in the Y direction. The connected shape of thephotoelectric conversion units 321 a to 321 f of the second pixel 321 isalso square. The outputs of the second pixel 321 are used for recordingimage generation and focus detection in a large defocus state (when thefocus shift amount is large). In FIG. 21, out of the six dividedphotoelectric conversion units, predetermined two photoelectricconversion units are used for focus detection. As described above withreference to FIGS. 11 to 13 of the first embodiment, the positions ofthe exit pupil of the photographing optical system projected onto thephotoelectric conversion units 321 a to 321 f change depending on theexit pupil distance of the photographing optical system and the imageheight (X-Y coordinates) of the pixel of interest. Hence, the eclipsedstate of each of the six photoelectric conversion units 321 a to 321 fis calculated in accordance with a predetermined equation. Twophotoelectric conversion units out of the photoelectric conversion units321 a to 321 f, which have the minimum eclipse, are used for focusdetection, thereby performing accurate focus detection. That is, in thesecond pixels 221 to 223 of the first embodiment, the photoelectricconversion units have three segmentation patterns, and optimumphotoelectric conversion units are selectively used in accordance withthe eclipsed state. By contrast, in the second embodiment, the secondpixel 321 has only one segmentation pattern, and the pixel array rule issimplified. Note that the pixels having the same structure and arrangedin the image sensor 107 will also be referred to as the second pixels321 hereinafter.

The second pixel 321 is a pixel for pupil division in the X directionand is used for an object having a luminance distribution in the Xdirection, that is, an object having a vertical-striped pattern. To thecontrary, a third pixel 322 is a pixel for pupil division in the Ydirection and is used for an object having a luminance distribution inthe Y direction, that is, an object having a lateral-striped pattern.The third pixel 322 includes six photoelectric conversion units 322 a to322 f arranged in the Y direction. The pattern is equivalent to that ofthe second pixel 321 rotated by 90°, and a detailed description thereofwill be omitted. The connection shape of the photoelectric conversionunits is square in all of the first to third pixels. A suitable one ofthe second pixel 321 and the third pixel 322 is used in accordance withthe luminance distribution pattern of a focus detection target object,thereby reducing the probability that focus detection is impossible.

FIGS. 22A to 22C conceptually illustrate a method of adding the outputsof the photoelectric conversion units and a pixel interpolation methodwhen creating a 3D image according to the second embodiment. FIGS. 22Ato 22C show the addition method for only a case in which the gravityacts in the Y-axis direction.

FIG. 22A shows the arrangements of the photoelectric conversion unitsprovided in the first pixel 311 and the second pixel 321 anew.

FIGS. 22B and 22C are views for explaining a pixel signal processingmethod when the user holds the camera in the landscape orientation, thatis, when the user holds the camera to make the negative direction of theY-axis of the image sensor shown in FIG. 21 match the direction ofgravity. FIG. 22B shows the addition method when the center of theprojected image of the exit pupil of the photographing optical systemdoes not decenter with respect to the connection center of thephotoelectric conversion units. In this case, for the first pixel 311,the sum of the signals of the photoelectric conversion units 311 a and311 b arranged on the left side serves as one signal of the parallaximage, and the sum of the signals of the photoelectric conversion units311 e and 311 f arranged on the right side serves as the other signal.The photoelectric conversion units 311 c and 311 d at the center are notused because they do not contribute to parallax formation.

For the second pixel 321, the sum of the signals of the photoelectricconversion units 321 a and 321 b to the left serves as one signal of theparallax image, and the sum of the signals of the photoelectricconversion units 321 e and 321 f to the right serves as the othersignal. The parallax information can also be obtained from the twophotoelectric conversion units 321 c and 321 d to the center. However,to make the parallax information have the same characteristics as thoseobtained from the first pixel 311, the signals of the two photoelectricconversion units 321 c and 321 d are not used.

The third pixel 322 has no parallax information in the horizontal axisdirection. Hence, the parallax signals are created by the sameinterpolation processing as that described with reference to FIGS. 16Ato 16C of the first embodiment, and a description thereof using thedrawings will be omitted.

FIG. 22C shows the addition method when the center of the projectedimage of the exit pupil of the photographing optical system decenters tothe left, that is, in the negative direction on the X-axis with respectto the connection center of the photoelectric conversion units. In thiscase, for the first pixel 311, the sum of the signals of thephotoelectric conversion units 311 a and 311 b arranged on the left sideserves as one signal of the parallax image, and the sum of the signalsof the photoelectric conversion units 311 c and 311 d arranged at thecenter serves as the other signal. The signals of the photoelectricconversion units 311 e and 311 f on the right side are not used becausesufficient amount of light does not enter due to vignetting.

For the second pixel 321, the sum of the signals of the photoelectricconversion units 321 a and 321 b to the left serves as one signal of theparallax image, and the sum of the signals of the photoelectricconversion units 321 c and 321 d to the center serves as the othersignal. The signals of the two photoelectric conversion units 321 e and321 f to the right are not used because sufficient amount of light doesnot enter due to vignetting. For the third pixel, the same processing asdescribed with reference to FIG. 22B is performed, and a descriptionthereof will be omitted.

Note that when the user holds the camera in the portrait orientation,that is, when the user holds the camera to make the negative directionof the X-axis of the image sensor 107 shown in FIG. 2 match thedirection of gravity, the addition processing of the photoelectricconversion units is performed in a direction orthogonal to the additiondirection in FIGS. 22B and 22C. The processing is the same as thatdescribed with reference to FIGS. 22B and 22C except that the secondpixel 321 and the third pixel 322 are used reversely, and a descriptionthereof will be omitted.

The relationship between the number m1 of X-direction segments of thephotoelectric conversion units in the first pixel 311 and the number n1of Y-direction segments is set to m1>n1 for the following reason.

In FIGS. 11 to 13 of the first embodiment, the decentering state of theprojected image of the exit pupil of the photographing optical systemwith respect to the connection center of the photoelectric conversionunits has been described. The decentering amount is proportional to thedeviation amount from the standard value of the exit pupil distance ofthe photographing optical system and the image height of the imagecapturing region. Since the image capturing region has a rectangularshape long in the X direction, the maximum image height has a largeX-direction component, and the maximum value of the decentering amountalso has a large X-direction component. Hence, increasing the number ofX-direction segments makes it possible to increase the degree of freedomof selection at the time of output addition and reliably acquire 3Dinformation at an arbitrary image height.

Increasing the number of segments of the photoelectric conversion unitsalso allows to increase the degree of combination at the time ofaddition. However, since the amount of image information also increases,the image processing apparatus is required to have high-speed processingperformance. Hence, in the second embodiment, the number of Y-directionsegments is set to 2, as in the first embodiment. The addition methodfor obtaining a 3D image from images photographed under the attitude inwhich the gravity acts in the X-axis direction is the same as thatdescribed with reference to FIG. 16C of the first embodiment, and adescription thereof will be omitted.

As described above, according to the second embodiment, the numbers ofsegments of the photoelectric conversion units of the first pixel in thetwo directions are made to match the direction dependence of thephotographing screen size. This allows to reliably obtain 3D informationfor arbitrary coordinates of the rectangular image capturing region andprevent the amount of image information from becoming excessively large.

Third Embodiment

In the above-described first and second embodiments, the numbers ofsegments of photoelectric conversion units are equal in the first tothird pixels. That is, the number of segments is four in the firstembodiment, and six in the second embodiment.

In the third embodiment, however, the number of segments ofphotoelectric conversion units is smaller in the second pixel than inthe first pixel. The pixels included in an image sensor 107 according tothe third embodiment will be described below with reference to FIG. 23.Note that since the arrangement is actually the same as in the first andsecond embodiments except the number of segments of photoelectricconversion units, only the different points will be described, and adescription of same or similar parts will be omitted.

FIG. 23 shows the pixel array of the image sensor 107 according to thethird embodiment of the present invention. Note that a description ofthe same ability as in FIGS. 2 and 21 will be omitted.

A first pixel 411 includes a total of six photoelectric conversion units411 a to 411 f segmented into three parts (integer m1=3) in the Xdirection and two parts (integer n1=2) in the Y direction, as in thesecond embodiment. The application purpose of the outputs of the firstpixel 411 is the same as that of the first pixel described in the firstand second embodiments. Note that the other pixels having the samestructure and arranged in the image sensor 107 will also be referred toas the first pixels 411 hereinafter.

Second pixels 421 are discretely arranged among the first pixels 411 inaccordance with a predetermined array rule. The second pixel 421includes a total of four photoelectric conversion units 421 a to 421 dsegmented into four parts (integer m2=4) in the X direction but notsegmented (integer n2=1) in the Y direction. Out of the photoelectricconversion units 421 a to 421 d of each second pixel 421, the regions ofthe two photoelectric conversion units at the center have a smallX-direction size, and the regions of the two outer photoelectricconversion units have a large X-direction size. The connected shape ofthe photoelectric conversion units 421 a to 421 d is square. Theapplication purpose of the outputs of the second pixel 421 is the sameas that of the second pixel described in the first and secondembodiments. Note that the pixels having the same structure and arrangedin the image sensor 107 will also be referred to as the second pixels421 hereinafter.

The second pixel 421 is a pixel for pupil division in the X directionand is used for an object having a luminance distribution in the Xdirection, that is, an object having a vertical-striped pattern. To thecontrary, a third pixel 422 is a pixel for pupil division in the Ydirection and is used for an object having a luminance distribution inthe Y direction, that is, an object having a lateral-striped pattern.The third pixel 422 also includes four photoelectric conversion units422 a to 422 d segmented in the Y direction. The pattern is equivalentto that of the second pixel 421 rotated by 90°, and a detaileddescription thereof will be omitted. The connection shape of thephotoelectric conversion units is square in all of the first to thirdpixels. A suitable one of the second pixel 421 and the third pixel 422is used in accordance with the luminance distribution pattern of a focusdetection target object, thereby reducing the probability that focusdetection is impossible.

The image sensor of the third embodiment is especially suitable when thechange in the exit pupil distance of the applied photographing opticalsystem is small. When the change in the exit pupil distance of thephotographing optical system, which occurs at the time of lens exchangeor the zoom operation, is small, the relative decentering amount betweenthe photoelectric conversion units and the exit pupil at the peripheryof the image capturing region described with reference to FIGS. 11 to 13of the first embodiment is also small. Hence, the signals for focusdetection can reliably be obtained even when the number of segments ofthe second pixel 421 is smaller. Decreasing the number of segmentsallows to decrease the amount of information to be read and furtherspeed up pixel signal read. In addition, since the pixel structure issimple, the manufacturing process becomes simple, and the variation inthe characteristics can effectively be reduced.

Note that in the third embodiment, a 3D image can be created basicallyusing the same addition method as that described with reference to FIGS.22A to 22C of the second embodiment, and a detailed description thereofwill be omitted.

As described above, according to the third embodiment, the number ofsegments of photoelectric conversion units is minimized in the secondand third pixels. This allows to decrease the information amount andthus speed up signal read. In addition, since the second and thirdpixels have a simple pixel structure, the yield in the manufacturingprocess can be improved, and the variation in the characteristics of theimage sensor can be reduced.

In the above-described first to third embodiments, the first pixel isdivided into two or three parts in the X direction and two parts in theY direction. However, the present invention is not limited to this. Thenumber of segments need not always be 2 or 3 and can be any integerlarger than 1.

In the above-described first to third embodiments, the number ofsegments of the second and third pixels is 4 or 6. However, the presentinvention is not limited to this. The number of segments only need be atleast twice the smaller one of the numbers of X- and Y-directionsegments of the first pixel.

In the above-described first to third embodiments, the image sensorincludes the first, second, and third pixels. However, to widen thefocus detectable defocus range and improve the detection accuracy in thenear in-focus state, the image sensor may be provided with only one ofthe second pixel group and the third pixel group, and the other groupmay be replaced with the first pixels. For example, when the thirdpixels are replaced with the first pixels, the irregularity of the pixelarray is reduced, resulting in a simpler image sensor structure. Inaddition, since the ratio of the first pixels rises, the number ofpixels to be interpolated upon creating a 3D image decreases, and theaccuracy of 3D information can be improved. On the other hand, theabsence of the third pixels may lower the focus detection capability ina largely blurred scene. However, since the final in-focus determinationcan be done using the first pixels, the in-focus accuracy does notlower.

Fourth Embodiment

In the above first to third embodiments, the segmentation pattern of thephotoelectric conversion units of each pixel and the method of selectingthe photoelectric conversion units at the time of focus detection havebeen described. The fourth embodiment to be described below isconfigured to improve the in-focus accuracy using an image sensor of thepresent invention.

FIG. 24 is a flowchart of the main procedure representing a procedure ofphotographing processing according to the fourth embodiment. FIG. 25 isa flowchart of a focus detection subroutine to be performed in step S431of FIG. 24. In the fourth embodiment, an image sensor 107 having thesame arrangement as that of the first embodiment described withreference to FIGS. 1 to 17B is used. However, the image sensor of thesecond or third embodiment may be used. The main procedure and the focusdetection subroutine of the photographing processing are partiallydifferent from the main procedure in FIG. 18 and the focus detectionsubroutine in FIG. 19 described in the first embodiment. Hence, the samestep numbers denote the same parts, and a detailed description thereofwill be omitted.

The main procedure of photographing processing will be described. In themain procedure shown in FIG. 24, the user turns on the power switch ofthe camera in step S101. In steps S102 to S109, a CPU 121 performsoperation check of the members in the camera, initialization,photographing condition setting, and the like. In step S431, the focusdetection subroutine shown in FIG. 25 is executed.

In step S461, in-focus flag state determination corresponding toin-focus determination is done. The in-focus flag represents whether thedefocus amount calculated by focus detection calculation is equal to orsmaller than a threshold at which the state can be regarded as in-focus.The in-focus flag can take three values “0”, “1”, and “2”. The in-focusflag is first initialized to “0”. As will be described later in thefocus detection subroutine shown in FIG. 25, when focus detection isperformed using a first pixel group 211, and the in-focus state isdetermined, “1” is stored in the in-focus flag. If focus detection isperformed using one of second pixel groups 221 to 223, and the in-focusstate is determined, “2” is stored in the in-focus flag. In the fourthembodiment, only when “1” is stored in the in-focus flag, the processcan advance from step S461 to step S153. On the other hand, the state ofthe in-focus flag is determined in step S461, and if the value isdetermined to be “0” or “2”, the focus lens is driven in step S462.Then, the process returns to the focus detection subroutine of step S431to repetitively perform focus detection.

If the in-focus flag is determined to be “1”, that is, the in-focusstate is determined in step S461, the process advances to step S153 todetermine whether the photographing switch is turned on. If thephotographing switch is turned on, image recording is executed in stepS161. If the photographing switch is not turned on, the process advancesto step S181 without executing step S161. In step S181, the CPU 121determines the state of the main switch. If the main switch remains on,steps S102 to S461 are repetitively executed. If the main switch is off,steps S182 to S184 are executed, and photographing ends.

The difference between the first embodiment and the fourth embodimentwill be described. In the first embodiment shown in FIG. 18, as can beseen from the flow of steps S131 to S153, even if the focus detectionresult indicates out-of-focus, the process directly advances to stepS153 after execution of focus lens driving. Hence, as far as thephotographing switch is turned on in step S153, the process can advanceto photographing, that is, the image recording subroutine of step S161even if the in-focus state is not guaranteed. To the contrary, in thefourth embodiment shown in FIG. 24, as can be seen from the flow ofsteps S431 to S153, if the focus detection result indicatesout-of-focus, the process returns to the focus detection subroutineafter execution of focus lens driving. As a result, the process advancesto the image recording subroutine of step S161 after the in-focus stateis confirmed in the focus detection subroutine.

The focus detection subroutine will be described with reference to FIG.25. In step S432, the object pattern is recognized from the previewimage, and face image determination, contrast analysis of the entirephotographing screen, and the like are performed. In step S433, the mainobject to be in focus is determined based on the recognition result instep S432. In step S434, the exit pupil of the photographing opticalsystem is calculated based on the lens information acquired in step S107of FIG. 24. More specifically, the size of the exit pupil and itsdistance from the image plane are calculated, and vignetting for eachimage height is calculated.

In step S435, the in-focus flag is determined. The in-focus flagrepresents the in-focus state at the time of in-focus operations, asdescribed concerning step S461 of FIG. 24. The in-focus flag is definedto store “0” if the in-focus state is not obtained, “1” if the in-focusstate is obtained using the first pixel group 211, and “2” if thein-focus state is obtained using one of the second pixel groups 221 to223 at the time of step S461. If step S435 is executed for the firsttime in the series of in-focus operations, the process advances to stepS436 because the in-focus flag has been initialized to “0”. In stepS436, a pixel group which is less affected by vignetting and suitablefor focus detection is selected in each focus detection area based onthe exit pupil information calculated in step S434. In this case, thepixel group is selected from the second pixel groups 221 to 223. Forexample, one of the second pixel groups 221, 222, and 223 is selected asdescribed with reference to FIGS. 11 to 13 used in the first embodiment.In step S437, a flag representing the type of the pixel group selectedfor focus detection is set. Since one of the second pixel groups 221 to223 is selected, the pixel flag is set to “2”. In step S440, a pair ofimages to be used for correlation is created using the outputs of twophotoelectric conversion units to the center in each pixel of theselected second pixel group.

In step S441, so-called shading correction is performed for the createdfocus detection signals to reduce the unbalance of the light amountscaused by vignetting. This allows to reduce the strength differencebetween two images and improve the focus detection accuracy. In stepS442, correlation is performed to calculate a lateral shift amount u ofthe two images that have undergone the shading correction. In step S443,the reliability of the image shift detection result is determined basedon the level of matching between the two images calculated in thecorrelation process of step S442. A value with a low reliability is notemployed.

In step S444, the defocus amount is calculated using equation (1) fromthe reliable image shift amount u obtained in steps S442 and S443 andbase-line lengths θ of the pixels used for focus detection.

In step S445, it is determined whether the defocus amount calculated instep S444 is equal to or smaller than an in-focus threshold. As thein-focus threshold, normally, the permissible depth on the image planeis employed. Let δ be the size of the permissible circle of confusion,and F be the f-number of the photographing optical system. In this case,a value calculated by F×δ is generally used as the permissible depth,that is, the in-focus threshold. Hence, if the defocus amount exceedsthe in-focus threshold, the process advances from step S445 to step S446to calculate the focus lens driving amount. The process then returnsfrom step S451 to the main routine.

The operation of the main routine after the return will be explained.When the process returns to the main routine shown in FIG. 24 afterexecution of the focus detection subroutine of step S431, the in-focusflag is determined in step S461. At this point of time, the in-focusflag is “0”. Hence, the focus lens is driven to cancel the detecteddefocus amount, and the focus detection subroutine of step S431 isexecuted again.

Upon determining in step S445 that the defocus amount is equal to orsmaller than the in-focus threshold, the process advances to step S447.In step S447, the pixel flag is determined. That is, the type of thepixel group used in the focus detection immediately before isdetermined. The pixel flag is “2” because one of the second pixel groups221 to 223 is selected in the first focus detection. Hence, the processadvances from step S447 to step S448 to set the in-focus flag to “2”.That is, it is determined at this point of time that the in-focus stateis obtained as the result of the in-focus operation using one of thesecond pixel groups 221 to 223. The process thus returns to the mainroutine via step S451.

The operation of the main routine after the return will be explainedagain. When the process returns to the main routine shown in FIG. 24after execution of the focus detection subroutine of step S431, thein-focus flag is determined in step S461. At this point of time, thein-focus flag is “2”. Hence, the process advances to step S462. In thiscase, the defocus amount is equal to or smaller than the in-focusthreshold, that is, a state regarded as in-focus has been obtained asthe result of focus detection using one of the second pixel groups 221to 223. Hence, the focus lens driving instruction is actually neglected.The process returns to step S431 to execute the focus detectionsubroutine again.

An operation of executing the focus detection subroutine when thein-focus flag is “2” will be described next. Steps S431 to S434 areexecuted, and the in-focus flag is determined in step S435. Since thein-focus flag is “2”, the process advances to step S438. In step S438,the first pixel group 211 is selected as the focus detection pixels. Instep S439, the flag representing the type of the pixel group selectedfor focus detection is set. Since the first pixel group 211 is selected,the pixel flag is set to “1”. In step S440, the outputs of twophotoelectric conversion units adjacent in the Y-axis direction areadded in each pixel of the first pixel group 211 to create a pair ofimages that divides the pupil in the X direction for correlation.

In step S441, so-called shading correction is performed for the createdfocus detection signals to reduce the unbalance of the light amountscaused by vignetting. This allows to reduce the strength differencebetween two images and improve the focus detection accuracy. In stepS442, correlation is performed to calculate the lateral shift amount uof the two images that have undergone the shading correction. In stepS443, the reliability of the image shift detection result is determinedbased on the level of matching between the two images calculated in thecorrelation process of step S442. A value with a low reliability is notemployed. In step S444, the defocus amount is calculated using equation(1) from the reliable image shift amount u obtained in steps S442 andS443 and the base-line lengths θ of the pixels used for focus detection.In step S445, it is determined whether the defocus amount calculated instep S444 is equal to or smaller than the in-focus threshold. If thedefocus amount exceeds the in-focus threshold, the process advances fromstep S445 to step S446 to calculate the focus lens driving amount. Theprocess then returns from step S451 to the main routine.

On the other hand, upon determining in step S445 that the defocus amountis equal to or smaller than the in-focus threshold, the process advancesto step S447. In step S447, the pixel flag is determined. That is, thetype of the pixel group used in the focus detection immediately beforeis determined. The pixel flag is “1” because the first pixel group 211is selected. Hence, the process advances from step S447 to step S449 toset the in-focus flag to “1”. That is, it is determined at this point oftime that the in-focus state is obtained as the result of the in-focusoperation using the first pixel group 211. In step S450, the defocus mapin the entire photographing region is created. The process returns tothe main routine via step S451.

The operation of the main routine after the return will be explained.When the process returns to the main routine shown in FIG. 24 afterexecution of the focus detection subroutine of step S431, the in-focusflag is determined in step S461. At this point of time, the in-focusflag is “1”. Hence, the process can advance to step S153. In step S153,the state of the photographing switch is determined. If the switch ison, the image recording subroutine is executed in step S161. Steps S182to S184 are further executed, and the photographing operation ends.

As described above, according to the fourth embodiment, focus detectionis performed first using the second pixel group with a small base-linelength in pupil division and also a small pupil width. Hence, focusdetection is possible even when the defocus amount is large in theinitial state. Next, focus detection is performed using the first pixelgroup with a large base-line length in pupil division. Hence, moreaccurate focus detection is possible. In the first pixel group, thepupil area at the time of focus detection is large, and a sufficientlight amount can be obtained. For this reason, accurate focus detectioncan be performed even for a low-luminance object.

In addition, since image recording is permitted after the in-focus stateis confirmed using the first pixel group, a high definition image infocus can be obtained.

Note that in the above-described fourth embodiment, one of the secondpixel groups 221 to 223 is used. However, if it is determined as theresult of object pattern recognition of step S432 that the object has aluminance difference in the vertical direction like a lateral-stripedpattern, third pixel groups 224 to 226 may be used in place of thesecond pixel groups 221 to 223. When one of the third pixel groups 224to 226 is used, “2” may be used as the pixel flag, or “3” may newly beset. In addition, “2” may be used as the in-focus flag, or “3” may newlybe set. In either case, the same processing is possible.

Fifth Embodiment

In the above-described fourth embodiment, focus detection is performedfirst using one of the second pixel groups 221 to 223 having a smallbase-line length, and then performed using the first pixel group 211having a large base-line length. In the fifth embodiment to be describedbelow, a plurality of focus detection results obtained using pixelgroups having different base-line lengths are multiplied by weightingcoefficients based on the reliability of the results and composited, ora reliable result is time-serially employed from a plurality of results.

FIGS. 26A and 26B explain a plurality of base-line lengths of an imagingpixel according to the fifth embodiment. An image sensor 107 used in thefifth embodiment has the same arrangement as that of the image sensor ofthe first embodiment except the photoelectric conversion unit selectionmethod in the focus detection procedure.

FIG. 26A shows the photoelectric conversion units of a first pixel 211.FIG. 26B shows the photoelectric conversion units of a second pixel 222.Referring to FIG. 26A, when performing focus detection by dividing thepupil in the X-axis direction, a pair of images for correlation iscreated using the sum signal of the photoelectric conversion unitscorresponding to pupil projected images EP1 a and EP1 b and the sumsignal of the photoelectric conversion units corresponding to pupilprojected images EP1 c and EP1 d. The base-line length at this time(more strictly, angle converted value viewed from the image plane) isθ2.

On the other hand, referring to FIG. 26B, a pair of images forcorrelation is created from the photoelectric conversion unitscorresponding to pupil projected images EP22 b and EP22 c. The base-linelength at this time is θ1. A pair of images for correlation can also becreated from the photoelectric conversion units corresponding to pupilprojected images EP22 a and EP22 d. The base-line length at this time isθ3. That is, in the fifth embodiment, three kinds of focus detectionsignals having different base-line lengths are used for focus detection.The base-line lengths of the signals hold a relationship represented byθ1<θ2<θ3  (5)Note that although FIG. 26B illustrates the photoelectric conversionunits of the second pixel 222, the first pixel 211 and a third pixel 225also have the same relationship except the pupil division direction, andthe output signals of these pixels can be processed like the secondpixel 222.

The advantages and disadvantages of the three kinds of focus detectionsignals and weighting coefficients set based on them will be describednext.

In general, when the base-line length in pupil division is large, therelative lateral shift amount of the pair of images for the unit defocusamount is large. Hence, the focus detection accuracy is high. On theother hand, when the base-line length is large, the lateral shift amountbetween the two images in the large defocus state is excessive, and theprobability that focus detection is impossible also rises. In addition,when the base-line length is large, a level difference readily occursbetween the two images due to vignetting of the photographing opticalsystem. A focus detection error occurs depending on the degree ofvignetting.

FIG. 27A shows weighting coefficients corresponding to f-numbers for theresults obtained from the three kinds of focus detection signals.DEF(θ1) to DEF(θ3) represent defocus amounts calculated from the focusdetection signals having the three kinds of base-line lengths describedwith reference to FIGS. 26A and 26B. F2.8 to F11 are f-numbers of thephotographing optical system at the time of focus detection. Numbers 0to 7 in the table represent weighting coefficients C(FN) for thecombinations of the defocus amounts and the f-numbers. In the fifthembodiment, three kinds of defocus amounts are obtained. The threeweighting coefficients corresponding to the defocus amounts are set suchthat the sum of the weighting coefficients is 10 for every f-number. Thesmaller the f-number is, that is, the larger the exit pupil of thephotographing optical system is, the larger the weighting coefficientfor a focus detection result with a large base-line length is.Conversely, the larger the f-number is, that is, the larger thevignetting of the photographing optical system is, the larger theweighting coefficient for a focus detection result with a smallbase-line length is.

FIG. 27B shows weighting coefficients corresponding to defocus amountsfor the results obtained from the three kinds of focus detectionsignals. In the fifth embodiment, the defocus amounts calculated fromthe focus detection signals are divided into four sections by themagnitude of the absolute value. The sections are shown in the uppermostline of the table. |Def| represents the absolute value of a detecteddefocus amount. Numbers 0 to 7 in the table represent weightingcoefficients C(DF) in the sections to which the three kinds of defocusamounts DEF(θ1) to DEF(θ3) belong. In FIG. 27B as well, the coefficientsare set such that the sum of the coefficients is 10 in each defocussection. When the defocus amount is small, the weighting of the resultobtained by photoelectric conversion units having a large base-linelength is set large. Conversely, since the larger the defocus amount is,the larger the relative lateral shift amount of two images is, theweighting of the focus detection result with a small base-line length isset large.

A final defocus amount DEF is calculated by multiplying the threedefocus amounts obtained from the pairs of focus detection signals withdifferent base-line lengths by the weighting coefficients defined in theabove-described manner using

$\begin{matrix}{{DEF} = {{{{DEF}\left( {\theta\; 1} \right)} \times C\; 1({FN}) \times C\; 1({DF})} + {{{DEF}\left( {\theta\; 2} \right)} \times C\; 2({FN}) \times C\; 2({DF})} + {{{DEF}\left( {\theta\; 3} \right)} \times C\; 3({FN}) \times C\; 3({DF})}}} & (6)\end{matrix}$Focus lens driving or in-focus determination is done based on thedefocus amount DEF.

FIG. 28 is a flowchart of a focus detection subroutine according to thefifth embodiment. Note that in the fifth embodiment, the same camera andimage sensor as those of the first to fourth embodiments are used. Themain procedure at the time of photographing of the camera is the same asin FIG. 24 described in the fourth embodiment, and a description of thesame parts will be omitted.

In the focus detection subroutine of FIG. 28, first, in step S532, theobject pattern is recognized from the preview image, and face imagedetermination, contrast analysis of the entire photographing screen, andthe like are performed. In step S533, the main object to be in focus isdetermined based on the recognition result in step S532. In step S534,the exit pupil of the photographing optical system is calculated basedon the lens information acquired in step S107 of FIG. 24. Morespecifically, the size of the exit pupil and its distance from the imageplane are calculated, and vignetting for each image height iscalculated.

In step S535, three sets of focus detection pixel groups that arepresent in the focus detection area are selected. The three setsindicate the photoelectric conversion unit output groups correspondingto the three kinds of base-line lengths described with reference toFIGS. 26A and 26B. In step S536, three pairs of images for correlationare created from the outputs of the photoelectric conversion units ofthe selected pixels.

In step S537, so-called shading correction is performed for the createdthree pairs of focus detection signals to reduce the unbalance of thelight amounts caused by vignetting. This allows to reduce the strengthdifference between two images and improve the focus detection accuracy.In step S538, correlation is performed to calculate a lateral shiftamount u of the two images that have undergone the shading correction.In step S539, the reliability of the image shift detection result isdetermined based on the level of matching between the two imagescalculated in the correlation process of step S538. In step S540, threedefocus amounts are calculated using equation (1) from the image shiftamount u obtained in step S538 and base-line lengths θ of the pixelsused for focus detection. In step S541, weighting by equation (6) isperformed for the three obtained defocus amounts, thereby obtaining thefinal defocus amount.

In step S542, it is determined whether the defocus amount calculated instep S541 is equal to or smaller than an in-focus threshold. If thedefocus amount exceeds the in-focus threshold, the process advances fromstep S542 to step S543 to calculate the focus lens driving amount. Theprocess then returns from step S546 to the main routine.

On the other hand, upon determining in step S542 that the defocus amountis equal to or smaller than the in-focus threshold, the process advancesto step S544 to set the in-focus flag to “1”. In step S545, the defocusmap is created. In step S546, the process returns to the main routine.

The operation of the main routine after the return will be explained.When the process returns to the main routine shown in FIG. 24 afterexecution of the focus detection subroutine of FIG. 28, the in-focusflag is determined in step S461. If the in-focus flag is not “1”, thatis, represents out-of-focus, the focus lens is driven in step S462.Then, the process returns to step S431 to execute the focus detectionsubroutine again.

On the other hand, upon determining in step S461 that the in-focus flagis “1”, the process advances to step S153 to perform image recording,image transmission, or the like, and the photographing operation ends.

As described above, according to the fifth embodiment, a plurality ofdefocus amounts are calculated from a plurality of kinds of focusdetection signals having different base-line lengths. The defocusamounts are weighted based on the photographing conditions such as thef-number of the photographing optical system or the focus state such asthe calculated defocus amounts, thereby calculating the final defocusamount. Since focus detection calculation can be done while emphasizingfocus detection signals having a base-line length suitable for thephotographing conditions and the focus state, accurate focus detectioncan be performed always.

Modifications of Fifth Embodiment

In the above-described fifth embodiment, a result obtained by weightinga plurality of focus detection signals is used as final focus detectioninformation. In modifications of the fifth embodiment to be describedbelow, a result assumed to be most reliable is alternatively selectedfrom a plurality of focus detection results.

FIG. 29 shows examples of weighting coefficients according to amodification and corresponds to FIG. 27A described above. In FIG. 27A,the weighting coefficients of the three kinds of focus detection signalsare finely set in accordance with the f-number of the photographingoptical system. In the modification shown in FIG. 29, one weightingcoefficient of the three kinds of signals is set to “1”, and theremaining two coefficients are set to “0”, thereby alternativelyselecting a result assumed to be most reliable. In this modification,since the alternative selection can be done before the phase differencedetection calculation of the pair of images, no wasteful calculationneed be performed. Hence, the focus detection calculation speeds up, andthe calculation program becomes simpler.

FIG. 30 shows examples of weighting coefficients according to anothermodification and corresponds to FIG. 27B described above. In FIG. 27B,the weighting coefficients of the three kinds of focus detection signalsare finely set in accordance with the calculated defocus amount. In themodification shown in FIG. 30, one weighting coefficient of the threekinds of signals is set to “1”, and the remaining two coefficients areset to “0”, thereby alternatively selecting a result assumed to be mostreliable. In this modification as well, since the weighting issimplified, the focus detection calculation speeds up, and thecalculation program becomes simpler.

In the fifth embodiment and the modifications thereof, the signals ofpixels corresponding to three kinds of base-line lengths are used. Usingone of them may be omitted, and only two desired kinds of signals may beused. Conversely, the arrangement may be applied to an embodimentincluding pixels corresponding to four or more kinds of base-linelengths. For example, in the example described above with reference toFIG. 26B, use of the photoelectric conversion units having the base-linelength θ3 in the second pixel group 221 may be prohibited.

Sixth Embodiment

In the above-described fourth embodiment, focus detection is performedfirst using the second pixel groups 221 to 223 having a small base-linelength, and then performed using the first pixel group 211 having alarge base-line length. In the sixth embodiment to be described below,this concept is expanded so that a plurality of focus detection resultsobtained with different base-line lengths are time-seriallyappropriately selected or composited even when the number of base-linelengths exceeds 2.

FIG. 31 shows examples of weighting coefficients according to the sixthembodiment. In the sixth embodiment, the weighting coefficients areswitched for the results obtained from the three kinds of focusdetection signals in accordance with the number of times of focusdetection.

DEF(θ1) to DEF(θ3) represent defocus amounts calculated from the focusdetection signals having the three kinds of base-line lengths describedwith reference to FIGS. 26A and 26B. In addition, “first time” to “thirdtime” represent the number of times of focus detection performed untilan in-focus state is obtained. Numbers 0 and 1 in the table representweighting coefficients C(SN) for the focus detection counts. A finaldefocus amount DEF is calculated by multiplying the three defocusamounts obtained from the pairs of focus detection signals withdifferent base-line lengths by the weighting coefficients defined in theabove-described manner using

$\begin{matrix}{{DEF} = {{{{DEF}\left( {\theta\; 1} \right)} \times C\; 1({SN})} + {{{DEF}\left( {\theta\; 2} \right)} \times C\; 2({SN})} + {{{DEF}\left( {\theta\; 3} \right)} \times C\; 3({SN})}}} & (7)\end{matrix}$In the sixth embodiment, the weighting coefficient is 0 or 1. Hence, apredetermined one of the plurality of pieces of focus detectioninformation is alternatively selected in focus detection of apredetermined time. That is, the focus detection signals by thebase-line length θ1 are selected in the first focus detection, the focusdetection signals by the base-line length θ2 are selected in the secondfocus detection, and the focus detection signals by the base-line lengthθ3 are selected in the third focus detection.

FIG. 32 is a flowchart of a focus detection subroutine according to thesixth embodiment. Note that in the sixth embodiment, the same camera andimage sensor as those of the first to fifth embodiments are used. Themain procedure at the time of photographing of the camera is the same asin FIG. 24 described in the fourth embodiment, and a description of thesame parts will be omitted.

In the focus detection subroutine of FIG. 32, first, in step S632, theobject pattern is recognized from the preview image, and face imagedetermination, contrast analysis of the entire photographing screen, andthe like are performed. In step S633, the main object to be in focus isdetermined based on the recognition result in step S632. In step S634,the exit pupil of the photographing optical system is calculated basedon the lens information acquired in step S107 of FIG. 24. Morespecifically, the size of the exit pupil and its distance from the imageplane are calculated, and vignetting for each image height iscalculated.

In step S635, three sets of focus detection pixel groups that arepresent in the focus detection area are selected. The three setsindicate the photoelectric conversion unit output groups correspondingto the three kinds of base-line lengths described with reference toFIGS. 26A and 26B. In step S636, three pairs of images for correlationare created from the outputs of the photoelectric conversion units ofthe selected pixels.

In step S637, so-called shading correction is performed for the createdthree pairs of focus detection signals to reduce the unbalance of thelight amounts caused by vignetting. This allows to reduce the strengthdifference between two images and improve the focus detection accuracy.In step S638, correlation is performed to calculate a lateral shiftamount u of the two images that have undergone the shading correction.In step S639, the reliability of the image shift detection result isdetermined based on the level of matching between the two imagescalculated in the correlation process of step S638. In step S640, threedefocus amounts are calculated using equation (1) from the image shiftamount u obtained in step S638 and base-line lengths θ of the pixelsused for focus detection. In step S641, an execution count N of focusdetection calculation until the in-focus state is obtained in the seriesof in-focus operations is recognized. The execution count is defined tobe 1 when all three sets of focus detection calculations for the threekinds of base-line lengths have been executed.

In step S642, weighting by equation (7) is performed for the threeobtained defocus amounts, thereby obtaining the final defocus amount.

In step S643, it is determined whether the defocus amount calculated instep S642 is equal to or smaller than an in-focus threshold. If thedefocus amount exceeds the in-focus threshold, the process advances fromstep S643 to step S644 to calculate the focus lens driving amount. Theprocess then returns from step S648 to the main routine.

On the other hand, upon determining in step S643 that the defocus amountis equal to or smaller than the in-focus threshold, the process advancesto step S645. In step S645, the value of the focus detection count N isdetermined. In the sixth embodiment, the process branches to YES when Nhas reached 3, and NO when N is 2 or less. That is, in the sixthembodiment, focus detection is performed for the three kinds ofbase-line lengths, and the detection results are employed sequentiallyfrom that obtained by signals with a small base-line length. Hence, whenthe three times of focus detection are completed, and the defocus amountis equal to or smaller than the in-focus threshold, the in-focus flag isset to “1” in step S646. In step S647, the defocus map is created. Instep S648, the process returns to the main routine.

The operation of the main routine after the return will be explainednext. When the process returns to the main routine shown in FIG. 24after execution of the focus detection subroutine of FIG. 32, thein-focus flag is determined in step S461. If the in-focus flag is not“1”, that is, represents out-of-focus, the focus lens is driven in stepS462. Then, the process returns to step S431 to execute the focusdetection subroutine again.

On the other hand, upon determining in step S461 that the in-focus flagis “1”, the process advances to step S153 to perform image recording,image transmission, or the like, and the photographing operation ends.

As described above, according to the sixth embodiment, focus detectionis performed first using pixels having a small base-line length, thenperformed using pixels having an intermediate base-line length, andfinally performed using pixels having a large base-line length in thehistory of focus detection up to the in-focus state. Since signalssuitable for each state from a large defocus amount to a small defocusamount are used, no wasteful calculation is performed, and accuratefocus detection is possible.

Modification of Sixth Embodiment

In the above-described sixth embodiment, predetermined signals arealternatively selected in accordance with the number of times (history)of focus detection. In the modification of the sixth embodiment to bedescribed below, weighting coefficients according to the history are setfor a plurality of focus detection results, and the plurality of resultsare composited.

FIG. 33 shows examples of weighting coefficients according to themodification and corresponds to FIG. 31 described above. In FIG. 31, apredetermined result is alternatively used in accordance with the focusdetection history. In the modification shown in FIG. 33, at least two ofthe three kinds of signals are used with predetermined weights. In thiscase, even if focus detection using a base-line length assumed to beoptimum is impossible, the defocus amount can be calculated if focusdetection using another base-line length is possible. This allows toprevent a phenomenon that the number of times of focus detection untilthe in-focus state is obtained unnecessarily increases.

In the sixth embodiment and the modification thereof, the signals ofpixels corresponding to three kinds of base-line lengths are used. Usingone of them may be omitted, and only two desired kinds of signals may beused. Conversely, the arrangement may be applied to an embodimentincluding pixels corresponding to four or more kinds of base-linelengths.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application Nos.2011-118397 filed on May 26, 2011 and 2012-045788 filed on Mar. 1, 2012,which are hereby incorporated by reference herein in their entirety.

What is claimed is:
 1. An image sensor comprising: a first imaging pixeland a second imaging pixel each of which detects an object image formedby a photographing optical system and generates a recording image,wherein said first imaging pixel comprises a plurality of photoelectricconversion units segmented in a first direction and said second imagingpixel includes four photoelectric conversion units segmented in thefirst direction, the plurality of photoelectric conversion units areconfigured to photoelectrically convert a plurality of images formed bysplit light beams passing through the photographing optical system andoutputting focus detection signals used to detect a phase difference,and a length between gravity centers of photoelectric conversion unitsused to detect the phase difference out of the plurality ofphotoelectric conversion units included in said first imaging pixel islonger than a length between gravity centers of photoelectric conversionunits used to detect the phase difference out of the four photoelectricconversion units included in said second imaging pixel.
 2. An imagesensor comprising: an imaging pixel which detects an object image formedby a photographing optical system and generates a recording image,wherein said imaging pixel comprises 2n photoelectric conversion unitssegmented in a first direction, where n is an integer not less than 2,the 2n photoelectric conversion units are configured tophotoelectrically convert 2n images formed by split light beams passingthrough the photographing optical system and outputting focus detectionsignals used to detect a phase difference, and the image sensor has amode in which the phase difference is detected using photoelectricconversion units of which a length between gravity centers is a firstlength out of the 2n photoelectric conversion units, and a mode in whichthe phase difference is detected using photoelectric conversion units ofwhich a length between gravity centers is a second length, which islonger than the first length, out of the 2n photoelectric conversionunits.
 3. An image capturing apparatus including an image sensoraccording to claim
 1. 4. An image capturing apparatus including an imagesensor according to claim 2.